[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:42:35 dist-test-slave-cr22 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:42:35 dist-test-slave-cr22 krb5kdc[399](info): set up 2 sockets
Jun 19 14:42:35 dist-test-slave-cr22 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 14:42:36.360641 392 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/test-user.keytab.
W20260619 14:42:36.737614 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.235s user 0.000s sys 0.002s
Jun 19 14:42:36 dist-test-slave-cr22 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:42:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:42:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260619 14:42:37.003566 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.240s user 0.000s sys 0.004s
I20260619 14:42:37.004563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43021
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:37.124949 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:37.125283 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:37.125396 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:37.129750 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:42:37.129860 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:37.129895 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:37.129936 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:42:37.129974 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:42:37.135707 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:37.136926 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:37.138131 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:37.148476 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:37.148437 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:37.148552 412 server_base.cc:1034] running on GCE node
W20260619 14:42:37.148437 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:37.149483 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:37.150724 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:37.151964 412 hybrid_clock.cc:648] HybridClock initialized: now 1781880157151854 us; error 126 us; skew 500 ppm
Jun 19 14:42:37 dist-test-slave-cr22 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:37.158593 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:42:37.159986 412 webserver.cc:460] Webserver started at http://127.0.98.62:41825/ using document root <none> and password file <none>
I20260619 14:42:37.161001 412 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:37.161104 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:37.161360 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:37.165566 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "f5508d909c9948f1badaae1d4ab190ae"
format_stamp: "Formatted at 2026-06-19 14:42:37 on dist-test-slave-cr22"
server_key: "b06bebccfe06e5235e404b3ff7136500"
server_key_iv: "609b5cd6d937c6749ed6ca04bab1af79"
server_key_version: "encryptionkey@0"
I20260619 14:42:37.166496 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "f5508d909c9948f1badaae1d4ab190ae"
format_stamp: "Formatted at 2026-06-19 14:42:37 on dist-test-slave-cr22"
server_key: "b06bebccfe06e5235e404b3ff7136500"
server_key_iv: "609b5cd6d937c6749ed6ca04bab1af79"
server_key_version: "encryptionkey@0"
I20260619 14:42:37.172844 412 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 14:42:37.176268 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:37.178007 412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:42:37.178184 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "f5508d909c9948f1badaae1d4ab190ae"
format_stamp: "Formatted at 2026-06-19 14:42:37 on dist-test-slave-cr22"
server_key: "b06bebccfe06e5235e404b3ff7136500"
server_key_iv: "609b5cd6d937c6749ed6ca04bab1af79"
server_key_version: "encryptionkey@0"
I20260619 14:42:37.178344 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:37.193374 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:37.197150 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:37.197425 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:37.205386 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43021
I20260619 14:42:37.205416 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43021 every 8 connection(s)
I20260619 14:42:37.206427 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:42:37.209452 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:37.212412 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 412
I20260619 14:42:37.212602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:42:37.214066 392 external_mini_cluster.cc:1428] Setting key 9a41c1e6d42ccf09746a6115dd394f2a
I20260619 14:42:37.215302 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae: Bootstrap starting.
I20260619 14:42:37.217876 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:37.218607 480 log.cc:824] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae: Log is configured to *not* fsync() on all Append() calls
I20260619 14:42:37.221252 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae: No bootstrap required, opened a new log
I20260619 14:42:37.225889 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5508d909c9948f1badaae1d4ab190ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43021 } }
I20260619 14:42:37.226145 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:37.226224 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5508d909c9948f1badaae1d4ab190ae, State: Initialized, Role: FOLLOWER
I20260619 14:42:37.226794 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5508d909c9948f1badaae1d4ab190ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43021 } }
I20260619 14:42:37.226958 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:42:37.227043 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:42:37.227152 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:37.228221 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5508d909c9948f1badaae1d4ab190ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43021 } }
I20260619 14:42:37.228567 480 leader_election.cc:302] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5508d909c9948f1badaae1d4ab190ae; no voters:
I20260619 14:42:37.228937 480 leader_election.cc:288] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:42:37.229008 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:42:37.229208 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae [term 1 LEADER]: Becoming Leader. State: Replica: f5508d909c9948f1badaae1d4ab190ae, State: Running, Role: LEADER
I20260619 14:42:37.229514 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5508d909c9948f1badaae1d4ab190ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43021 } }
I20260619 14:42:37.230257 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:42:37.232988 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5508d909c9948f1badaae1d4ab190ae. Latest consensus state: current_term: 1 leader_uuid: "f5508d909c9948f1badaae1d4ab190ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5508d909c9948f1badaae1d4ab190ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43021 } } }
I20260619 14:42:37.233263 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae [sys.catalog]: This master's current role is: LEADER
I20260619 14:42:37.233115 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5508d909c9948f1badaae1d4ab190ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5508d909c9948f1badaae1d4ab190ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43021 } } }
I20260619 14:42:37.233428 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae [sys.catalog]: This master's current role is: LEADER
Jun 19 14:42:37 dist-test-slave-cr22 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260619 14:42:37.236323 500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:42:37.236413 500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:42:37.236518 501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:42:37.237394 501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:42:37.242422 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:37.217313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54488 (local address 127.0.98.62:43021)
0619 14:42:37.217869 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:37.217890 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:37.218922 (+ 1032us) server_negotiation.cc:407] Connection header received
0619 14:42:37.223655 (+ 4733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:37.223679 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:37.224119 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:37.224497 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:37.227021 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:37.228166 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:37.229689 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:37.230209 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:37.230350 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:37.236302 (+ 5952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:37.236311 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:37.236327 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:37.236361 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:37.238659 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:37.239315 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:37.239322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:37.239330 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:37.239498 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:37.239859 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:37.239864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:37.239866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:37.240278 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:37.240495 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:37.241049 (+ 554us) server_negotiation.cc:299] Negotiation successful
0619 14:42:37.241327 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":158,"threads_started":1}
I20260619 14:42:37.243618 501 catalog_manager.cc:1347] Generated new cluster ID: 70da8e92859947bb97b94d2f52a5c41b
I20260619 14:42:37.243676 501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:42:37.278478 501 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:42:37.279539 501 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:42:37.289652 501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5508d909c9948f1badaae1d4ab190ae: Generated new TSK 0
I20260619 14:42:37.290472 501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:42:37.335649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43021
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:37.453824 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:37.454092 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:37.454145 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:37.458299 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:37.458367 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:37.458427 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:42:37.464470 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43021
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:37.465626 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:37.466702 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:37.477471 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:37.477519 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:37.477509 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:37.477716 506 server_base.cc:1034] running on GCE node
I20260619 14:42:37.478235 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:37.478945 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:37.480243 506 hybrid_clock.cc:648] HybridClock initialized: now 1781880157480212 us; error 45 us; skew 500 ppm
Jun 19 14:42:37 dist-test-slave-cr22 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:37.483903 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:42:37.484841 506 webserver.cc:460] Webserver started at http://127.0.98.1:39725/ using document root <none> and password file <none>
I20260619 14:42:37.485426 506 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:37.485492 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:37.485664 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:37.487824 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "3e30243d861f4126b34c1f803739981a"
format_stamp: "Formatted at 2026-06-19 14:42:37 on dist-test-slave-cr22"
server_key: "b74a7a8b4051cc15356cfb7f9c796f0b"
server_key_iv: "1111341c644d16d1ac54848db6afbda7"
server_key_version: "encryptionkey@0"
I20260619 14:42:37.488386 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3e30243d861f4126b34c1f803739981a"
format_stamp: "Formatted at 2026-06-19 14:42:37 on dist-test-slave-cr22"
server_key: "b74a7a8b4051cc15356cfb7f9c796f0b"
server_key_iv: "1111341c644d16d1ac54848db6afbda7"
server_key_version: "encryptionkey@0"
I20260619 14:42:37.493140 506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 14:42:37.496145 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:37.497998 506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:42:37.498112 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "3e30243d861f4126b34c1f803739981a"
format_stamp: "Formatted at 2026-06-19 14:42:37 on dist-test-slave-cr22"
server_key: "b74a7a8b4051cc15356cfb7f9c796f0b"
server_key_iv: "1111341c644d16d1ac54848db6afbda7"
server_key_version: "encryptionkey@0"
I20260619 14:42:37.498271 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:37.535728 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:37.540210 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:37.540445 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:37.541188 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:42:37.542179 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:37.542239 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:37.542291 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:37.542310 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:37.556324 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36379
I20260619 14:42:37.556676 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36379 every 8 connection(s)
I20260619 14:42:37.557360 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:42:37.563840 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 506
I20260619 14:42:37.563993 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:42:37.564276 392 external_mini_cluster.cc:1428] Setting key 9d6050a16a7be63f1f46d155b6534521
Jun 19 14:42:37 dist-test-slave-cr22 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:42:37.576438 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:37.559387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54051 (local address 127.0.98.62:43021)
0619 14:42:37.559588 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:37.559592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:37.560448 (+ 856us) server_negotiation.cc:407] Connection header received
0619 14:42:37.561909 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:37.561913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:37.561972 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:37.562053 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:37.563261 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:37.563951 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:37.565530 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:37.565841 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:37.565976 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:37.569934 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:37.569939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:37.569942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:37.569978 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:37.572231 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:37.573077 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:37.573082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:37.573084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:37.573180 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:37.574208 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:37.574215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:37.574217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:37.574603 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:37.574762 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:37.576027 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0619 14:42:37.576202 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:42:37.577538 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:37.559792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43021 (local address 127.0.98.1:54051)
0619 14:42:37.560275 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:37.560319 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:37.561664 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:37.562230 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:37.562237 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:37.562614 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:37.563111 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:37.563124 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:37.564096 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:37.564100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:37.565388 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:37.565399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:37.566199 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:37.566204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:37.566413 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:37.567096 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:37.567123 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:37.569593 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:37.572492 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:37.572499 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:37.572518 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:37.572872 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:37.573295 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:37.573298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:37.573300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:37.573526 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:37.574782 (+ 1256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:37.574795 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:37.575604 (+ 809us) client_negotiation.cc:769] Sending connection context
0619 14:42:37.575946 (+ 342us) client_negotiation.cc:240] Negotiation successful
0619 14:42:37.576387 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":110,"threads_started":1}
I20260619 14:42:37.579547 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43021
I20260619 14:42:37.580006 635 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:37.580597 635 heartbeater.cc:511] Master 127.0.98.62:43021 requested a full tablet report, sending...
I20260619 14:42:37.582553 444 ts_manager.cc:194] Registered new tserver with Master: 3e30243d861f4126b34c1f803739981a (127.0.98.1:36379)
I20260619 14:42:37.584100 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:42:37.597061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43021
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:37.714188 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:37.714546 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:37.714653 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:37.720028 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:37.720136 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:37.720358 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:42:37.726817 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43021
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:37.728384 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:37.729609 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:37.741103 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:37.741107 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:37.741441 640 server_base.cc:1034] running on GCE node
W20260619 14:42:37.741526 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:37.742033 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:37.742830 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:37.744030 640 hybrid_clock.cc:648] HybridClock initialized: now 1781880157744010 us; error 31 us; skew 500 ppm
Jun 19 14:42:37 dist-test-slave-cr22 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:37.746997 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:42:37.748032 640 webserver.cc:460] Webserver started at http://127.0.98.2:34887/ using document root <none> and password file <none>
I20260619 14:42:37.748624 640 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:37.748690 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:37.748907 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:37.751005 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "993526252e594ffb853ebe2c5bcb44ac"
format_stamp: "Formatted at 2026-06-19 14:42:37 on dist-test-slave-cr22"
server_key: "c06f8dae460e4c5cf47fa696ac59c3e1"
server_key_iv: "d1abef1d99b281b9a4b99071e3c13d75"
server_key_version: "encryptionkey@0"
I20260619 14:42:37.751662 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "993526252e594ffb853ebe2c5bcb44ac"
format_stamp: "Formatted at 2026-06-19 14:42:37 on dist-test-slave-cr22"
server_key: "c06f8dae460e4c5cf47fa696ac59c3e1"
server_key_iv: "d1abef1d99b281b9a4b99071e3c13d75"
server_key_version: "encryptionkey@0"
I20260619 14:42:37.755882 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:42:37.758556 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:37.760037 640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:42:37.760166 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "993526252e594ffb853ebe2c5bcb44ac"
format_stamp: "Formatted at 2026-06-19 14:42:37 on dist-test-slave-cr22"
server_key: "c06f8dae460e4c5cf47fa696ac59c3e1"
server_key_iv: "d1abef1d99b281b9a4b99071e3c13d75"
server_key_version: "encryptionkey@0"
I20260619 14:42:37.760290 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:37.778752 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:37.782608 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:37.782845 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:37.783679 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:42:37.784585 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:37.784636 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:37.784709 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:37.784755 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:37.795619 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36349
I20260619 14:42:37.795640 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36349 every 8 connection(s)
I20260619 14:42:37.796608 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:42:37.805856 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 640
I20260619 14:42:37.805985 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:42:37.806247 392 external_mini_cluster.cc:1428] Setting key ea45a7846c246676de558cbc8673e9cb
Jun 19 14:42:37 dist-test-slave-cr22 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:37.811728 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:37.798536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54723 (local address 127.0.98.62:43021)
0619 14:42:37.798679 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:37.798682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:37.799595 (+ 913us) server_negotiation.cc:407] Connection header received
0619 14:42:37.801050 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:37.801053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:37.801107 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:37.801217 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:37.802448 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:37.802972 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:37.804226 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:37.804455 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:37.804566 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:37.807860 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:37.807864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:37.807867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:37.807896 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:37.809410 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:37.810051 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:37.810054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:37.810055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:37.810131 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:37.810536 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:37.810538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:37.810539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:37.810710 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:37.810793 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:37.811487 (+ 694us) server_negotiation.cc:299] Negotiation successful
0619 14:42:37.811603 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:42:37.812995 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:37.798963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43021 (local address 127.0.98.2:54723)
0619 14:42:37.799396 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:37.799485 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:37.800837 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:37.801395 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:37.801402 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:37.801750 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:37.802311 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:37.802326 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:37.803104 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:37.803112 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:37.804103 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:37.804112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:37.804602 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:37.804606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:37.805173 (+ 567us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:37.805742 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:37.805840 (+ 98us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:37.807660 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:37.809520 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:37.809528 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:37.809544 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:37.809907 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:37.810223 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:37.810229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:37.810234 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:37.810431 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:37.810814 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:37.810823 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:37.811151 (+ 328us) client_negotiation.cc:769] Sending connection context
0619 14:42:37.811357 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 14:42:37.811746 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":104,"threads_started":1}
I20260619 14:42:37.814376 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43021
I20260619 14:42:37.814647 769 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:37.815176 769 heartbeater.cc:511] Master 127.0.98.62:43021 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:42:37.816555 444 ts_manager.cc:194] Registered new tserver with Master: 993526252e594ffb853ebe2c5bcb44ac (127.0.98.2:36349)
I20260619 14:42:37.817219 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:42:37.840333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43021
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:37.963395 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:37.963778 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:37.963831 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:37.968750 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:37.968865 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:37.968982 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:42:37.975677 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43021
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:37.976910 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:37.978001 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:37.988791 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:37.988791 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:37.988929 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:37.989113 774 server_base.cc:1034] running on GCE node
I20260619 14:42:37.989578 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:37.990175 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:37.991374 774 hybrid_clock.cc:648] HybridClock initialized: now 1781880157991332 us; error 53 us; skew 500 ppm
Jun 19 14:42:37 dist-test-slave-cr22 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:37.995047 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:42:37.996145 774 webserver.cc:460] Webserver started at http://127.0.98.3:40027/ using document root <none> and password file <none>
I20260619 14:42:37.996724 774 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:37.996798 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:37.996974 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:37.999256 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "df5d83fb8826419181edc0300f36047d"
format_stamp: "Formatted at 2026-06-19 14:42:37 on dist-test-slave-cr22"
server_key: "27cd1511091f462bfdd5e9039c71e27e"
server_key_iv: "cda2c4e96b65d13536e03893915daab2"
server_key_version: "encryptionkey@0"
I20260619 14:42:37.999905 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "df5d83fb8826419181edc0300f36047d"
format_stamp: "Formatted at 2026-06-19 14:42:37 on dist-test-slave-cr22"
server_key: "27cd1511091f462bfdd5e9039c71e27e"
server_key_iv: "cda2c4e96b65d13536e03893915daab2"
server_key_version: "encryptionkey@0"
I20260619 14:42:38.005393 774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:42:38.008495 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:38.010015 774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:42:38.010187 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "df5d83fb8826419181edc0300f36047d"
format_stamp: "Formatted at 2026-06-19 14:42:37 on dist-test-slave-cr22"
server_key: "27cd1511091f462bfdd5e9039c71e27e"
server_key_iv: "cda2c4e96b65d13536e03893915daab2"
server_key_version: "encryptionkey@0"
I20260619 14:42:38.010344 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:38.026592 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:38.030871 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:38.031133 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:38.031968 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:42:38.033398 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:38.033557 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:38.033634 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:38.033697 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:38.046037 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43117
I20260619 14:42:38.046092 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43117 every 8 connection(s)
I20260619 14:42:38.047125 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:42:38.048799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 774
I20260619 14:42:38.048951 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:42:38.049279 392 external_mini_cluster.cc:1428] Setting key 0de73f3b23356c01d7ffc329b65bc854
Jun 19 14:42:38 dist-test-slave-cr22 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:38.064512 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:38.049580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55575 (local address 127.0.98.62:43021)
0619 14:42:38.049739 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:38.049743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:38.050567 (+ 824us) server_negotiation.cc:407] Connection header received
0619 14:42:38.052258 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:38.052261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:38.052326 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:38.052408 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:38.053614 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:38.054150 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:38.055330 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:38.055654 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:38.055768 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:38.058693 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:38.058697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:38.058702 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:38.058733 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:38.060916 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:38.061752 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:38.061755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:38.061757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:38.061876 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:38.062473 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:38.062477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:38.062479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:38.062777 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:38.062890 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:38.064180 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0619 14:42:38.064328 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:42:38.065707 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:38.049846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43021 (local address 127.0.98.3:55575)
0619 14:42:38.050404 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:38.050464 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:38.052001 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:38.052581 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:38.052589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:38.052973 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:38.053466 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:38.053479 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:38.054297 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:38.054300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:38.055194 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:38.055203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:38.055816 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:38.055820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:38.055996 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:38.056602 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:38.056619 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:38.058500 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:38.061137 (+ 2637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:38.061148 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:38.061171 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:38.061601 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:38.061979 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:38.061983 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:38.061986 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:38.062219 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:38.062906 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:38.062916 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:38.063831 (+ 915us) client_negotiation.cc:769] Sending connection context
0619 14:42:38.064131 (+ 300us) client_negotiation.cc:240] Negotiation successful
0619 14:42:38.064534 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":167,"threads_started":1}
I20260619 14:42:38.067239 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43021
I20260619 14:42:38.067555 903 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:38.068110 903 heartbeater.cc:511] Master 127.0.98.62:43021 requested a full tablet report, sending...
I20260619 14:42:38.069216 444 ts_manager.cc:194] Registered new tserver with Master: df5d83fb8826419181edc0300f36047d (127.0.98.3:43117)
I20260619 14:42:38.069868 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55575
I20260619 14:42:38.078351 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:42:38 dist-test-slave-cr22 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880158, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:42:38 dist-test-slave-cr22 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880158, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:42:38.113154 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:38.099560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34404 (local address 127.0.98.1:36379)
0619 14:42:38.099914 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:38.099920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:38.099949 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:42:38.100057 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:38.100062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:38.100327 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:38.100458 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:38.101305 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:38.102532 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:38.103978 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:38.104397 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:38.104564 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:38.107694 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:38.107704 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:38.107717 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:38.107745 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:38.110393 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:38.111121 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:38.111126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:38.111127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:38.111222 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:38.111757 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:38.111762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:38.111764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:38.112241 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:38.112459 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:38.112730 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 14:42:38.113010 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":106,"threads_started":1}
W20260619 14:42:38.114249 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34404
Jun 19 14:42:38 dist-test-slave-cr22 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:42:38 dist-test-slave-cr22 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:42:38.145913 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:38.134576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.0.98.1:36379)
0619 14:42:38.134933 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:38.134938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:38.134965 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:42:38.135034 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:38.135037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:38.135206 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:38.135363 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:38.136496 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:38.137057 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:38.138169 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:38.138423 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:38.138512 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:38.142020 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:38.142023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:38.142026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:38.142054 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:38.143732 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:38.144185 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:38.144188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:38.144189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:38.144263 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:38.144506 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:38.144508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:38.144509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:38.144678 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:38.144791 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:38.145334 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 14:42:38.145729 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260619 14:42:38.147615 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 506
I20260619 14:42:38.154536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 640
I20260619 14:42:38.162989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 774
I20260619 14:42:38.168710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 412
2026-06-19T14:42:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2902 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:42:38 dist-test-slave-cr22 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:42:38 dist-test-slave-cr22 krb5kdc[917](info): set up 2 sockets
Jun 19 14:42:38 dist-test-slave-cr22 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260619 14:42:39.259487 392 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:42:39 dist-test-slave-cr22 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:42:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:42:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:42:39.396072 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38489
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:39.509886 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:39.510164 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:39.510207 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:39.514590 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:42:39.514664 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:39.514685 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:39.514703 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:42:39.514717 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:42:39.520438 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:39.522083 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:39.523284 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:39.534088 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:39.534077 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:39.534081 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:39.535086 930 server_base.cc:1034] running on GCE node
I20260619 14:42:39.535919 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:39.537472 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:39.538722 930 hybrid_clock.cc:648] HybridClock initialized: now 1781880159538702 us; error 35 us; skew 500 ppm
Jun 19 14:42:39 dist-test-slave-cr22 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:39.542263 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:42:39.543280 930 webserver.cc:460] Webserver started at http://127.0.98.62:44765/ using document root <none> and password file <none>
I20260619 14:42:39.543934 930 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:39.544027 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:39.544263 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:39.546432 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "b04c08494e7d4813acd1bd20818a0862"
format_stamp: "Formatted at 2026-06-19 14:42:39 on dist-test-slave-cr22"
server_key: "3d5945ce5e65ed580bf2d0e3a5dbc51d"
server_key_iv: "d06cecfa68fa9aa70c9d7f0acdba7ec8"
server_key_version: "encryptionkey@0"
I20260619 14:42:39.547013 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "b04c08494e7d4813acd1bd20818a0862"
format_stamp: "Formatted at 2026-06-19 14:42:39 on dist-test-slave-cr22"
server_key: "3d5945ce5e65ed580bf2d0e3a5dbc51d"
server_key_iv: "d06cecfa68fa9aa70c9d7f0acdba7ec8"
server_key_version: "encryptionkey@0"
I20260619 14:42:39.551569 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260619 14:42:39.554906 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:39.556548 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:42:39.556744 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "b04c08494e7d4813acd1bd20818a0862"
format_stamp: "Formatted at 2026-06-19 14:42:39 on dist-test-slave-cr22"
server_key: "3d5945ce5e65ed580bf2d0e3a5dbc51d"
server_key_iv: "d06cecfa68fa9aa70c9d7f0acdba7ec8"
server_key_version: "encryptionkey@0"
I20260619 14:42:39.556902 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:39.577693 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:39.581877 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:39.582118 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:39.589915 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38489
I20260619 14:42:39.589954 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38489 every 8 connection(s)
I20260619 14:42:39.590915 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:42:39.593317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 930
I20260619 14:42:39.593469 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:42:39.593745 392 external_mini_cluster.cc:1428] Setting key 17736fe4744fc77221d8fac98ff1ef37
I20260619 14:42:39.594173 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:39.601359 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862: Bootstrap starting.
Jun 19 14:42:39 dist-test-slave-cr22 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:39.604180 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:39.604982 998 log.cc:824] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862: Log is configured to *not* fsync() on all Append() calls
I20260619 14:42:39.608319 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862: No bootstrap required, opened a new log
I20260619 14:42:39.611558 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:39.595191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.0.98.62:38489)
0619 14:42:39.595662 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:39.595678 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:39.595718 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:42:39.596484 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:39.596503 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:39.596869 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:39.597243 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:39.598059 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:39.599147 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:39.600430 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:39.600970 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:39.601166 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:39.604747 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:39.604755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:39.604778 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:39.604825 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:39.607254 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:39.608040 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:39.608048 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:39.608055 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:39.608396 (+ 341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:39.608844 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:39.608848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:39.608850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:39.609441 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:39.609661 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:39.609954 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 14:42:39.610316 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":119,"threads_started":1}
I20260619 14:42:39.613309 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04c08494e7d4813acd1bd20818a0862" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38489 } }
I20260619 14:42:39.613610 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:39.613663 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b04c08494e7d4813acd1bd20818a0862, State: Initialized, Role: FOLLOWER
I20260619 14:42:39.614171 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04c08494e7d4813acd1bd20818a0862" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38489 } }
I20260619 14:42:39.614354 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:42:39.614428 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:42:39.614508 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:39.615613 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04c08494e7d4813acd1bd20818a0862" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38489 } }
I20260619 14:42:39.615939 998 leader_election.cc:302] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b04c08494e7d4813acd1bd20818a0862; no voters:
I20260619 14:42:39.616225 998 leader_election.cc:288] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:42:39.616365 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:42:39.616614 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862 [term 1 LEADER]: Becoming Leader. State: Replica: b04c08494e7d4813acd1bd20818a0862, State: Running, Role: LEADER
I20260619 14:42:39.616994 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04c08494e7d4813acd1bd20818a0862" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38489 } }
I20260619 14:42:39.617591 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:42:39.618844 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b04c08494e7d4813acd1bd20818a0862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04c08494e7d4813acd1bd20818a0862" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38489 } } }
I20260619 14:42:39.619073 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862 [sys.catalog]: This master's current role is: LEADER
I20260619 14:42:39.619426 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b04c08494e7d4813acd1bd20818a0862. Latest consensus state: current_term: 1 leader_uuid: "b04c08494e7d4813acd1bd20818a0862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04c08494e7d4813acd1bd20818a0862" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38489 } } }
I20260619 14:42:39.619594 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862 [sys.catalog]: This master's current role is: LEADER
I20260619 14:42:39.620344 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:42:39.624737 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:42:39.631237 1012 catalog_manager.cc:1347] Generated new cluster ID: ea83607db2204e53a76eafc342516bc6
I20260619 14:42:39.631327 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:42:39.639014 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:42:39.640331 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:42:39.658491 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b04c08494e7d4813acd1bd20818a0862: Generated new TSK 0
I20260619 14:42:39.659358 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:42:39.704607 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38489
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:39.833664 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:39.834008 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:39.834092 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:39.838408 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:39.838534 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:39.838640 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:42:39.844794 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38489
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:39.846001 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:39.847075 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:39.857201 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:39.857203 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:39.857321 1024 server_base.cc:1034] running on GCE node
W20260619 14:42:39.857203 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:39.857931 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:39.858521 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:39.859776 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781880159859740 us; error 49 us; skew 500 ppm
Jun 19 14:42:39 dist-test-slave-cr22 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:39.863021 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:42:39.864017 1024 webserver.cc:460] Webserver started at http://127.0.98.1:33977/ using document root <none> and password file <none>
I20260619 14:42:39.864594 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:39.864686 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:39.864934 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:39.867101 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "962c3576ba2d4b3693807bb9e9288c30"
format_stamp: "Formatted at 2026-06-19 14:42:39 on dist-test-slave-cr22"
server_key: "d34d6ab4b50e58c76870c31d10544b20"
server_key_iv: "58efca8a9ec2e1d0611fcd3ecde6f668"
server_key_version: "encryptionkey@0"
I20260619 14:42:39.867733 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "962c3576ba2d4b3693807bb9e9288c30"
format_stamp: "Formatted at 2026-06-19 14:42:39 on dist-test-slave-cr22"
server_key: "d34d6ab4b50e58c76870c31d10544b20"
server_key_iv: "58efca8a9ec2e1d0611fcd3ecde6f668"
server_key_version: "encryptionkey@0"
I20260619 14:42:39.872102 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:42:39.874990 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:39.876845 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:42:39.877017 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "962c3576ba2d4b3693807bb9e9288c30"
format_stamp: "Formatted at 2026-06-19 14:42:39 on dist-test-slave-cr22"
server_key: "d34d6ab4b50e58c76870c31d10544b20"
server_key_iv: "58efca8a9ec2e1d0611fcd3ecde6f668"
server_key_version: "encryptionkey@0"
I20260619 14:42:39.877148 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:39.900776 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:39.905686 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:39.906013 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:39.906814 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:42:39.907842 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:39.907917 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:39.907994 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:39.908038 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:39.923242 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46553
I20260619 14:42:39.923273 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46553 every 8 connection(s)
I20260619 14:42:39.924643 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:42:39.932968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 1024
I20260619 14:42:39.933121 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:42:39.933411 392 external_mini_cluster.cc:1428] Setting key f967409e9f2472ed425ae9373a7e610a
Jun 19 14:42:39 dist-test-slave-cr22 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:39.940469 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:39.927144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45391 (local address 127.0.98.62:38489)
0619 14:42:39.927327 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:39.927331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:39.928341 (+ 1010us) server_negotiation.cc:407] Connection header received
0619 14:42:39.929751 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:39.929755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:39.929814 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:39.929923 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:39.931161 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:39.931952 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:39.933382 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:39.933682 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:39.933817 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:39.936677 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:39.936681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:39.936684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:39.936715 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:39.938257 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:39.938859 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:39.938863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:39.938865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:39.938942 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:39.939277 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:39.939281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:39.939282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:39.939520 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:39.939626 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:39.940164 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 14:42:39.940288 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 14:42:39.941262 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:39.927608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38489 (local address 127.0.98.1:45391)
0619 14:42:39.928107 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:39.928162 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:39.929527 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:39.930116 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:39.930122 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:39.930527 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:39.931015 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:39.931027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:39.932098 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:39.932102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:39.933112 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:39.933127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:39.933860 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:39.933865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:39.934046 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:39.934704 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:39.934724 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:39.936492 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:39.938447 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:39.938461 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:39.938477 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:39.938747 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:39.939033 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:39.939038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:39.939040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:39.939175 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:39.939637 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:39.939645 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:39.939928 (+ 283us) client_negotiation.cc:769] Sending connection context
0619 14:42:39.940115 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 14:42:39.940450 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:42:39.942611 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38489
I20260619 14:42:39.942931 1153 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:39.943475 1153 heartbeater.cc:511] Master 127.0.98.62:38489 requested a full tablet report, sending...
I20260619 14:42:39.945175 962 ts_manager.cc:194] Registered new tserver with Master: 962c3576ba2d4b3693807bb9e9288c30 (127.0.98.1:46553)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:42:39.946422 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:42:39.963423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38489
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:40.080876 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:40.081171 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:40.081236 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:40.085525 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:40.085592 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:40.085654 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:42:40.090996 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38489
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:40.092193 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:40.093202 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:40.105504 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:40.105401 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:40.105401 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:40.106027 1158 server_base.cc:1034] running on GCE node
I20260619 14:42:40.106545 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:40.107197 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:40.110499 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781880160110463 us; error 64 us; skew 500 ppm
Jun 19 14:42:40 dist-test-slave-cr22 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:40.113751 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:42:40.114889 1158 webserver.cc:460] Webserver started at http://127.0.98.2:44415/ using document root <none> and password file <none>
I20260619 14:42:40.115576 1158 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:40.115680 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:40.116119 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:40.118672 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "6503abf781cd4d0896334814fe589f2a"
format_stamp: "Formatted at 2026-06-19 14:42:40 on dist-test-slave-cr22"
server_key: "abdd78d0245905ba04e40157124c7d6b"
server_key_iv: "4b1cfd8765ec6ac9d2b338bffd0e66e2"
server_key_version: "encryptionkey@0"
I20260619 14:42:40.119491 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6503abf781cd4d0896334814fe589f2a"
format_stamp: "Formatted at 2026-06-19 14:42:40 on dist-test-slave-cr22"
server_key: "abdd78d0245905ba04e40157124c7d6b"
server_key_iv: "4b1cfd8765ec6ac9d2b338bffd0e66e2"
server_key_version: "encryptionkey@0"
I20260619 14:42:40.124220 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:42:40.127836 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:40.130002 1158 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:42:40.130198 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "6503abf781cd4d0896334814fe589f2a"
format_stamp: "Formatted at 2026-06-19 14:42:40 on dist-test-slave-cr22"
server_key: "abdd78d0245905ba04e40157124c7d6b"
server_key_iv: "4b1cfd8765ec6ac9d2b338bffd0e66e2"
server_key_version: "encryptionkey@0"
I20260619 14:42:40.130353 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:40.145856 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:40.149227 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:40.149439 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:40.150046 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:42:40.150877 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:40.150949 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:40.151026 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:40.151068 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:40.160271 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32973
I20260619 14:42:40.160382 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32973 every 8 connection(s)
I20260619 14:42:40.161331 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:42:40.170923 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 1158
Jun 19 14:42:40 dist-test-slave-cr22 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:40.171113 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:42:40.171428 392 external_mini_cluster.cc:1428] Setting key 81f752fa0e732f902ece2b7d38665741
I20260619 14:42:40.177124 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:40.163281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38263 (local address 127.0.98.62:38489)
0619 14:42:40.163510 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:40.163515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:40.164423 (+ 908us) server_negotiation.cc:407] Connection header received
0619 14:42:40.165825 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:40.165829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:40.165883 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:40.165967 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:40.167117 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.167726 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.168888 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.169111 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.169223 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:40.171765 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:40.171768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:40.171770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:40.171793 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:40.173685 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:40.174218 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:40.174224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:40.174228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:40.174318 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:40.175016 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:40.175024 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:40.175029 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:40.175555 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:40.175720 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:40.176779 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0619 14:42:40.176949 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:42:40.178071 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:40.163728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38489 (local address 127.0.98.2:38263)
0619 14:42:40.164243 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:40.164282 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:40.165617 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:40.166150 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:40.166156 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:40.166503 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:40.166979 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:40.166991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.167869 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.167875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:40.168758 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:40.168767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.169259 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.169264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:40.169430 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:40.170038 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:40.170054 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:40.171586 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:40.173819 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:40.173825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:40.173838 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:40.174106 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:40.174521 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:40.174526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:40.174527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:40.174798 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:40.175734 (+ 936us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:40.175751 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:40.176387 (+ 636us) client_negotiation.cc:769] Sending connection context
0619 14:42:40.176683 (+ 296us) client_negotiation.cc:240] Negotiation successful
0619 14:42:40.177100 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":124,"threads_started":1}
I20260619 14:42:40.179900 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38489
I20260619 14:42:40.180188 1287 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:40.180776 1287 heartbeater.cc:511] Master 127.0.98.62:38489 requested a full tablet report, sending...
I20260619 14:42:40.182171 962 ts_manager.cc:194] Registered new tserver with Master: 6503abf781cd4d0896334814fe589f2a (127.0.98.2:32973)
I20260619 14:42:40.183136 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38263
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:42:40.207770 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38489
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:40.338145 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:40.338413 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:40.338456 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:40.342588 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:40.342657 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:40.342720 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:42:40.348201 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38489
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:40.349264 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:40.350263 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:40.360644 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:40.360818 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:40.361097 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:40.361164 1292 server_base.cc:1034] running on GCE node
I20260619 14:42:40.361559 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:40.362321 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:40.363535 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781880160363530 us; error 48 us; skew 500 ppm
Jun 19 14:42:40 dist-test-slave-cr22 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:40.366458 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:42:40.367344 1292 webserver.cc:460] Webserver started at http://127.0.98.3:38965/ using document root <none> and password file <none>
I20260619 14:42:40.367949 1292 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:40.368013 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:40.368178 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:40.370319 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "0c7aca83028447b9a76c29cebd90ddb7"
format_stamp: "Formatted at 2026-06-19 14:42:40 on dist-test-slave-cr22"
server_key: "5f272c0ea038939a26bbb2a1c81f82bf"
server_key_iv: "cf90ec71ef0f160ce726344eedab900a"
server_key_version: "encryptionkey@0"
I20260619 14:42:40.371061 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0c7aca83028447b9a76c29cebd90ddb7"
format_stamp: "Formatted at 2026-06-19 14:42:40 on dist-test-slave-cr22"
server_key: "5f272c0ea038939a26bbb2a1c81f82bf"
server_key_iv: "cf90ec71ef0f160ce726344eedab900a"
server_key_version: "encryptionkey@0"
I20260619 14:42:40.374948 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:42:40.377658 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:40.379066 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:42:40.379166 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "0c7aca83028447b9a76c29cebd90ddb7"
format_stamp: "Formatted at 2026-06-19 14:42:40 on dist-test-slave-cr22"
server_key: "5f272c0ea038939a26bbb2a1c81f82bf"
server_key_iv: "cf90ec71ef0f160ce726344eedab900a"
server_key_version: "encryptionkey@0"
I20260619 14:42:40.379251 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:40.393567 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:40.397490 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:40.397670 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:40.398218 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:42:40.399040 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:40.399097 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:40.399142 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:40.399160 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:40.408867 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39555
I20260619 14:42:40.408936 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39555 every 8 connection(s)
I20260619 14:42:40.409811 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:42:40.416121 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 1292
I20260619 14:42:40.416330 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:42:40.416702 392 external_mini_cluster.cc:1428] Setting key 750d06248a12b9b00c91988be235a895
Jun 19 14:42:40 dist-test-slave-cr22 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:40.429180 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:40.411926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39043 (local address 127.0.98.62:38489)
0619 14:42:40.412085 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:40.412089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:40.412947 (+ 858us) server_negotiation.cc:407] Connection header received
0619 14:42:40.414423 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:40.414427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:40.414494 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:40.414611 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:40.415854 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.416529 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.418239 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.418469 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.418592 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:40.422026 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:40.422032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:40.422035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:40.422066 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:40.426124 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:40.426891 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:40.426895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:40.426897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:40.427059 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:40.427592 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:40.427599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:40.427604 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:40.427887 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:40.428008 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:40.428845 (+ 837us) server_negotiation.cc:299] Negotiation successful
0619 14:42:40.428998 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 14:42:40.430449 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:40.412282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38489 (local address 127.0.98.3:39043)
0619 14:42:40.412788 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:40.412832 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:40.414191 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:40.414797 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:40.414804 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:40.415173 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:40.415698 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:40.415712 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.416740 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.416743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:40.418068 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:40.418077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.418626 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.418630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:40.418886 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:40.419645 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:40.419663 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:40.421797 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:40.426323 (+ 4526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:40.426332 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:40.426354 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:40.426776 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:40.427178 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:40.427183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:40.427186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:40.427396 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:40.428015 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:40.428023 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:40.428543 (+ 520us) client_negotiation.cc:769] Sending connection context
0619 14:42:40.428761 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 14:42:40.429141 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":100,"threads_started":1}
I20260619 14:42:40.431864 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38489
I20260619 14:42:40.432201 1421 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:40.432861 1421 heartbeater.cc:511] Master 127.0.98.62:38489 requested a full tablet report, sending...
I20260619 14:42:40.434486 962 ts_manager.cc:194] Registered new tserver with Master: 0c7aca83028447b9a76c29cebd90ddb7 (127.0.98.3:39555)
I20260619 14:42:40.435410 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39043
I20260619 14:42:40.444140 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:42:40.460409 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:40.448863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36284 (local address 127.0.98.62:38489)
0619 14:42:40.449029 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:40.449036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:40.449159 (+ 123us) server_negotiation.cc:407] Connection header received
0619 14:42:40.449353 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:40.449357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:40.449413 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:40.449492 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:40.450621 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.451682 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.452719 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.453036 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.453151 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:40.454857 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:40.454861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:40.454864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:40.454898 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:40.457676 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:40.458519 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:40.458525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:40.458528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:40.458647 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:40.459134 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:40.459139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:40.459141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:40.459431 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:40.459630 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:40.460030 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 14:42:40.460193 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:42:40.467986 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:42:40.471663 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:42:40.495576 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:40.489391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39555 (local address 127.0.0.1:34800)
0619 14:42:40.490341 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:40.490355 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:40.490477 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:40.491143 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:40.491146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:40.491168 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:40.491282 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:40.491289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.492567 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.492571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:40.494107 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:40.494118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.494942 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.494945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:40.495048 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:40.495082 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 14:42:40.495226 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 14:42:40.495354 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":112,"thread_start_us":111,"threads_started":1}
I20260619 14:42:40.495640 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:40.489905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34800 (local address 127.0.98.3:39555)
0619 14:42:40.490287 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:40.490305 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:40.490590 (+ 285us) server_negotiation.cc:407] Connection header received
0619 14:42:40.490669 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:40.490674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:40.490840 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:40.490962 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:40.491415 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.492437 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.494256 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.494813 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.494941 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:40.495093 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:40.495258 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 14:42:40.495354 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":175,"threads_started":1}
I20260619 14:42:40.495879 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:40.488956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.0.98.1:46553)
0619 14:42:40.489397 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:40.489410 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:40.490451 (+ 1041us) server_negotiation.cc:407] Connection header received
0619 14:42:40.490542 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:40.490549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:40.490737 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:40.490905 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:40.491194 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.492530 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.494111 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.494941 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.495056 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:40.495186 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:40.495633 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 14:42:40.495720 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
I20260619 14:42:40.495878 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:40.488939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.0.98.2:32973)
0619 14:42:40.489373 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:40.489387 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:40.489912 (+ 525us) server_negotiation.cc:407] Connection header received
0619 14:42:40.490217 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:40.490224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:40.490447 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:40.490613 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:40.491549 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.492735 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.494234 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.494974 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.495062 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:40.495185 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:40.495624 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 14:42:40.495719 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":148,"threads_started":1}
I20260619 14:42:40.496276 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:40.488664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46553 (local address 127.0.0.1:60406)
0619 14:42:40.490334 (+ 1670us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:40.490347 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:40.490435 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:40.490909 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:40.490915 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:40.490936 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:40.491075 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:40.491084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.492624 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.492627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:40.493979 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:40.493991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.495250 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.495253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:40.495396 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:40.495477 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 14:42:40.496018 (+ 541us) client_negotiation.cc:240] Negotiation successful
0619 14:42:40.496113 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1610,"mutex_wait_us":1359,"thread_start_us":104,"threads_started":1}
I20260619 14:42:40.496778 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:40.488664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32973 (local address 127.0.0.1:38162)
0619 14:42:40.489753 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:40.489822 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:40.490025 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:40.491288 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:40.491290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:40.491305 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:40.491406 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:40.491411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.492874 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.492877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:40.494108 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:40.494118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.495250 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.495253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:40.495422 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:40.495493 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 14:42:40.496567 (+ 1074us) client_negotiation.cc:240] Negotiation successful
0619 14:42:40.496662 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":105,"thread_start_us":73,"threads_started":1}
I20260619 14:42:40.498984 1221 tablet_service.cc:1467] Processing CreateTablet for tablet fe47c3212226496c92c51a92afa3b003 (DEFAULT_TABLE table=test-table [id=38de42ac66274249afaee5ba53da1528]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:40.498860 1355 tablet_service.cc:1467] Processing CreateTablet for tablet fe47c3212226496c92c51a92afa3b003 (DEFAULT_TABLE table=test-table [id=38de42ac66274249afaee5ba53da1528]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:40.498857 1087 tablet_service.cc:1467] Processing CreateTablet for tablet fe47c3212226496c92c51a92afa3b003 (DEFAULT_TABLE table=test-table [id=38de42ac66274249afaee5ba53da1528]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:40.499953 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe47c3212226496c92c51a92afa3b003. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:40.499971 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe47c3212226496c92c51a92afa3b003. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:40.500789 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe47c3212226496c92c51a92afa3b003. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:40.506018 1438 tablet_bootstrap.cc:492] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a: Bootstrap starting.
I20260619 14:42:40.507880 1439 tablet_bootstrap.cc:492] T fe47c3212226496c92c51a92afa3b003 P 962c3576ba2d4b3693807bb9e9288c30: Bootstrap starting.
I20260619 14:42:40.508801 1438 tablet_bootstrap.cc:654] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:40.510588 1439 tablet_bootstrap.cc:654] T fe47c3212226496c92c51a92afa3b003 P 962c3576ba2d4b3693807bb9e9288c30: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:40.510741 1438 log.cc:824] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:42:40.511519 1439 log.cc:824] T fe47c3212226496c92c51a92afa3b003 P 962c3576ba2d4b3693807bb9e9288c30: Log is configured to *not* fsync() on all Append() calls
I20260619 14:42:40.513266 1438 tablet_bootstrap.cc:492] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a: No bootstrap required, opened a new log
I20260619 14:42:40.513499 1438 ts_tablet_manager.cc:1397] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 14:42:40.513677 1439 tablet_bootstrap.cc:492] T fe47c3212226496c92c51a92afa3b003 P 962c3576ba2d4b3693807bb9e9288c30: No bootstrap required, opened a new log
I20260619 14:42:40.514084 1439 ts_tablet_manager.cc:1397] T fe47c3212226496c92c51a92afa3b003 P 962c3576ba2d4b3693807bb9e9288c30: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:42:40.518726 1438 raft_consensus.cc:348] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503abf781cd4d0896334814fe589f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32973 } } peers { permanent_uuid: "962c3576ba2d4b3693807bb9e9288c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } } peers { permanent_uuid: "0c7aca83028447b9a76c29cebd90ddb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39555 } }
I20260619 14:42:40.511585 1440 tablet_bootstrap.cc:492] T fe47c3212226496c92c51a92afa3b003 P 0c7aca83028447b9a76c29cebd90ddb7: Bootstrap starting.
I20260619 14:42:40.519209 1439 raft_consensus.cc:348] T fe47c3212226496c92c51a92afa3b003 P 962c3576ba2d4b3693807bb9e9288c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503abf781cd4d0896334814fe589f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32973 } } peers { permanent_uuid: "962c3576ba2d4b3693807bb9e9288c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } } peers { permanent_uuid: "0c7aca83028447b9a76c29cebd90ddb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39555 } }
I20260619 14:42:40.519492 1439 raft_consensus.cc:374] T fe47c3212226496c92c51a92afa3b003 P 962c3576ba2d4b3693807bb9e9288c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:40.519598 1439 raft_consensus.cc:729] T fe47c3212226496c92c51a92afa3b003 P 962c3576ba2d4b3693807bb9e9288c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 962c3576ba2d4b3693807bb9e9288c30, State: Initialized, Role: FOLLOWER
I20260619 14:42:40.520141 1439 consensus_queue.cc:260] T fe47c3212226496c92c51a92afa3b003 P 962c3576ba2d4b3693807bb9e9288c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503abf781cd4d0896334814fe589f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32973 } } peers { permanent_uuid: "962c3576ba2d4b3693807bb9e9288c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } } peers { permanent_uuid: "0c7aca83028447b9a76c29cebd90ddb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39555 } }
I20260619 14:42:40.520959 1153 heartbeater.cc:503] Master 127.0.98.62:38489 was elected leader, sending a full tablet report...
I20260619 14:42:40.521322 1439 ts_tablet_manager.cc:1428] T fe47c3212226496c92c51a92afa3b003 P 962c3576ba2d4b3693807bb9e9288c30: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260619 14:42:40.521754 1440 tablet_bootstrap.cc:654] T fe47c3212226496c92c51a92afa3b003 P 0c7aca83028447b9a76c29cebd90ddb7: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:40.523132 1440 log.cc:824] T fe47c3212226496c92c51a92afa3b003 P 0c7aca83028447b9a76c29cebd90ddb7: Log is configured to *not* fsync() on all Append() calls
I20260619 14:42:40.519079 1438 raft_consensus.cc:374] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:40.523591 1438 raft_consensus.cc:729] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503abf781cd4d0896334814fe589f2a, State: Initialized, Role: FOLLOWER
I20260619 14:42:40.524155 1438 consensus_queue.cc:260] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503abf781cd4d0896334814fe589f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32973 } } peers { permanent_uuid: "962c3576ba2d4b3693807bb9e9288c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } } peers { permanent_uuid: "0c7aca83028447b9a76c29cebd90ddb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39555 } }
I20260619 14:42:40.525292 1438 ts_tablet_manager.cc:1428] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260619 14:42:40.526557 1440 tablet_bootstrap.cc:492] T fe47c3212226496c92c51a92afa3b003 P 0c7aca83028447b9a76c29cebd90ddb7: No bootstrap required, opened a new log
I20260619 14:42:40.526925 1440 ts_tablet_manager.cc:1397] T fe47c3212226496c92c51a92afa3b003 P 0c7aca83028447b9a76c29cebd90ddb7: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20260619 14:42:40.527796 1287 heartbeater.cc:503] Master 127.0.98.62:38489 was elected leader, sending a full tablet report...
I20260619 14:42:40.532409 1440 raft_consensus.cc:348] T fe47c3212226496c92c51a92afa3b003 P 0c7aca83028447b9a76c29cebd90ddb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503abf781cd4d0896334814fe589f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32973 } } peers { permanent_uuid: "962c3576ba2d4b3693807bb9e9288c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } } peers { permanent_uuid: "0c7aca83028447b9a76c29cebd90ddb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39555 } }
I20260619 14:42:40.532823 1440 raft_consensus.cc:374] T fe47c3212226496c92c51a92afa3b003 P 0c7aca83028447b9a76c29cebd90ddb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:40.532915 1440 raft_consensus.cc:729] T fe47c3212226496c92c51a92afa3b003 P 0c7aca83028447b9a76c29cebd90ddb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c7aca83028447b9a76c29cebd90ddb7, State: Initialized, Role: FOLLOWER
I20260619 14:42:40.533614 1440 consensus_queue.cc:260] T fe47c3212226496c92c51a92afa3b003 P 0c7aca83028447b9a76c29cebd90ddb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503abf781cd4d0896334814fe589f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32973 } } peers { permanent_uuid: "962c3576ba2d4b3693807bb9e9288c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } } peers { permanent_uuid: "0c7aca83028447b9a76c29cebd90ddb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39555 } }
I20260619 14:42:40.534718 1421 heartbeater.cc:503] Master 127.0.98.62:38489 was elected leader, sending a full tablet report...
I20260619 14:42:40.534921 1440 ts_tablet_manager.cc:1428] T fe47c3212226496c92c51a92afa3b003 P 0c7aca83028447b9a76c29cebd90ddb7: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
W20260619 14:42:40.660825 1422 tablet.cc:2369] T fe47c3212226496c92c51a92afa3b003 P 0c7aca83028447b9a76c29cebd90ddb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:42:40.662663 1288 tablet.cc:2369] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:42:40.668303 1445 raft_consensus.cc:482] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:42:40.668491 1445 raft_consensus.cc:504] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503abf781cd4d0896334814fe589f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32973 } } peers { permanent_uuid: "962c3576ba2d4b3693807bb9e9288c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } } peers { permanent_uuid: "0c7aca83028447b9a76c29cebd90ddb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39555 } }
I20260619 14:42:40.669807 1445 leader_election.cc:288] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 962c3576ba2d4b3693807bb9e9288c30 (127.0.98.1:46553), 0c7aca83028447b9a76c29cebd90ddb7 (127.0.98.3:39555)
I20260619 14:42:40.673992 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:40.670109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39555 (local address 127.0.98.2:43671)
0619 14:42:40.670232 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:40.670249 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:40.670394 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:40.670775 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:40.670778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:40.670799 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:40.670939 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:40.670946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.671944 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.671947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:40.672791 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:40.672800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.673545 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.673549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:40.673681 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:40.673711 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:42:40.673782 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 14:42:40.673865 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 14:42:40.674088 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:40.670201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43671 (local address 127.0.98.3:39555)
0619 14:42:40.670319 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:40.670323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:40.670338 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:42:40.670574 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:40.670579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:40.670646 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:40.670734 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:40.671112 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.671810 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.672930 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.673419 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.673502 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:40.673555 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:40.673844 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 14:42:40.673924 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:42:40.674249 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:40.670109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46553 (local address 127.0.98.2:49441)
0619 14:42:40.670556 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:40.670571 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:40.670681 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:40.671112 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:40.671115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:40.671134 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:40.671253 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:40.671259 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.672117 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.672120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:40.672986 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:40.672995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.673826 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.673829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:40.673926 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:40.673948 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:42:40.674027 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 14:42:40.674095 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":48,"thread_start_us":91,"threads_started":1}
I20260619 14:42:40.674502 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:40.670370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49441 (local address 127.0.98.1:46553)
0619 14:42:40.670530 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:40.670536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:40.670686 (+ 150us) server_negotiation.cc:407] Connection header received
0619 14:42:40.670776 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:40.670778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:40.670836 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:40.670927 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:40.671398 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.671965 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.673115 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.673611 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.673716 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:40.673792 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:40.674072 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 14:42:40.674151 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:42:40.674798 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe47c3212226496c92c51a92afa3b003" candidate_uuid: "6503abf781cd4d0896334814fe589f2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c7aca83028447b9a76c29cebd90ddb7" is_pre_election: true
I20260619 14:42:40.675130 1375 raft_consensus.cc:2454] T fe47c3212226496c92c51a92afa3b003 P 0c7aca83028447b9a76c29cebd90ddb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6503abf781cd4d0896334814fe589f2a in term 0.
I20260619 14:42:40.675508 1105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe47c3212226496c92c51a92afa3b003" candidate_uuid: "6503abf781cd4d0896334814fe589f2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962c3576ba2d4b3693807bb9e9288c30" is_pre_election: true
I20260619 14:42:40.675796 1105 raft_consensus.cc:2454] T fe47c3212226496c92c51a92afa3b003 P 962c3576ba2d4b3693807bb9e9288c30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6503abf781cd4d0896334814fe589f2a in term 0.
I20260619 14:42:40.675819 1177 leader_election.cc:302] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c7aca83028447b9a76c29cebd90ddb7, 6503abf781cd4d0896334814fe589f2a; no voters:
W20260619 14:42:40.676183 1154 tablet.cc:2369] T fe47c3212226496c92c51a92afa3b003 P 962c3576ba2d4b3693807bb9e9288c30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:42:40.676167 1445 raft_consensus.cc:2781] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:42:40.676258 1445 raft_consensus.cc:482] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:42:40.676312 1445 raft_consensus.cc:3037] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:40.677378 1445 raft_consensus.cc:504] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503abf781cd4d0896334814fe589f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32973 } } peers { permanent_uuid: "962c3576ba2d4b3693807bb9e9288c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } } peers { permanent_uuid: "0c7aca83028447b9a76c29cebd90ddb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39555 } }
I20260619 14:42:40.677749 1445 leader_election.cc:288] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a [CANDIDATE]: Term 1 election: Requested vote from peers 962c3576ba2d4b3693807bb9e9288c30 (127.0.98.1:46553), 0c7aca83028447b9a76c29cebd90ddb7 (127.0.98.3:39555)
I20260619 14:42:40.678264 1105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe47c3212226496c92c51a92afa3b003" candidate_uuid: "6503abf781cd4d0896334814fe589f2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962c3576ba2d4b3693807bb9e9288c30"
I20260619 14:42:40.678272 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe47c3212226496c92c51a92afa3b003" candidate_uuid: "6503abf781cd4d0896334814fe589f2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c7aca83028447b9a76c29cebd90ddb7"
I20260619 14:42:40.678411 1105 raft_consensus.cc:3037] T fe47c3212226496c92c51a92afa3b003 P 962c3576ba2d4b3693807bb9e9288c30 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:40.678403 1375 raft_consensus.cc:3037] T fe47c3212226496c92c51a92afa3b003 P 0c7aca83028447b9a76c29cebd90ddb7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:40.679534 1105 raft_consensus.cc:2454] T fe47c3212226496c92c51a92afa3b003 P 962c3576ba2d4b3693807bb9e9288c30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6503abf781cd4d0896334814fe589f2a in term 1.
I20260619 14:42:40.679601 1375 raft_consensus.cc:2454] T fe47c3212226496c92c51a92afa3b003 P 0c7aca83028447b9a76c29cebd90ddb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6503abf781cd4d0896334814fe589f2a in term 1.
I20260619 14:42:40.679941 1174 leader_election.cc:302] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6503abf781cd4d0896334814fe589f2a, 962c3576ba2d4b3693807bb9e9288c30; no voters:
I20260619 14:42:40.680415 1445 raft_consensus.cc:2781] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:42:40.680665 1445 raft_consensus.cc:686] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a [term 1 LEADER]: Becoming Leader. State: Replica: 6503abf781cd4d0896334814fe589f2a, State: Running, Role: LEADER
I20260619 14:42:40.681109 1445 consensus_queue.cc:237] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503abf781cd4d0896334814fe589f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32973 } } peers { permanent_uuid: "962c3576ba2d4b3693807bb9e9288c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } } peers { permanent_uuid: "0c7aca83028447b9a76c29cebd90ddb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39555 } }
I20260619 14:42:40.684646 959 catalog_manager.cc:5495] T fe47c3212226496c92c51a92afa3b003 P 6503abf781cd4d0896334814fe589f2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6503abf781cd4d0896334814fe589f2a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6503abf781cd4d0896334814fe589f2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503abf781cd4d0896334814fe589f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "962c3576ba2d4b3693807bb9e9288c30" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c7aca83028447b9a76c29cebd90ddb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39555 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:42:40 dist-test-slave-cr22 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:42:40.745323 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:40.732657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60420 (local address 127.0.98.1:46553)
0619 14:42:40.732837 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:40.732842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:40.733385 (+ 543us) server_negotiation.cc:407] Connection header received
0619 14:42:40.733600 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:40.733603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:40.733663 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:40.733772 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:40.734535 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.735124 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.736188 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.736451 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.736579 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:40.740748 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:40.740761 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:40.740776 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:40.740805 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:40.743219 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:40.743846 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:40.743853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:40.743854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:40.743964 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:40.744265 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:40.744268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:40.744270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:40.744557 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:40.744883 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:40.744989 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 14:42:40.745128 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 14:42:40 dist-test-slave-cr22 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880160, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:42:40 dist-test-slave-cr22 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880160, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:42:40.778517 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:40.766417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60432 (local address 127.0.98.1:46553)
0619 14:42:40.766587 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:40.766591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:40.766606 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:42:40.766916 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:40.766920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:40.767130 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:40.767229 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:40.768065 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.768678 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.769853 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.770110 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.770232 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:40.773775 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:40.773778 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:40.773780 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:40.773807 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:40.776142 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:40.776807 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:40.776809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:40.776811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:40.776894 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:40.777319 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:40.777324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:40.777325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:40.777626 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:40.777775 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:40.778136 (+ 361us) server_negotiation.cc:299] Negotiation successful
0619 14:42:40.778307 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
Jun 19 14:42:40 dist-test-slave-cr22 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:42:40 dist-test-slave-cr22 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:42:40.809110 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:40.799918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.0.98.1:46553)
0619 14:42:40.800084 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:40.800087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:40.800108 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:42:40.800255 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:40.800258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:40.800301 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:40.800372 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:40.801047 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.801562 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.802433 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:40.802636 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:40.802723 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:40.805758 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:40.805761 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:40.805763 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:40.805785 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:40.807256 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:40.807791 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:40.807796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:40.807798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:40.807900 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:40.808188 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:40.808191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:40.808192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:40.808374 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:40.808486 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:40.808714 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 14:42:40.808875 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 14:42:40.810549 1067 tablet_service.cc:2908] Rejecting scan request for tablet fe47c3212226496c92c51a92afa3b003: Uninitialized: clean time has not yet been initialized
I20260619 14:42:40.813375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 1024
I20260619 14:42:40.821238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 1158
I20260619 14:42:40.827265 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 1292
I20260619 14:42:40.832837 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 930
2026-06-19T14:42:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2665 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:42:40 dist-test-slave-cr22 krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:42:40 dist-test-slave-cr22 krb5kdc[1462](info): set up 2 sockets
Jun 19 14:42:40 dist-test-slave-cr22 krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260619 14:42:41.911284 392 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:42:41 dist-test-slave-cr22 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:42:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:42:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260619 14:42:42.438671 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.422s user 0.001s sys 0.003s
I20260619 14:42:42.439218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41483
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:42.562458 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:42.562762 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:42.562816 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:42.567018 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:42:42.567085 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:42:42.567103 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:42.567121 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:42.567138 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:42:42.567153 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:42:42.572629 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41483
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:42.573724 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:42.574713 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:42.584442 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:42.584470 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:42.584456 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:42.584518 1475 server_base.cc:1034] running on GCE node
I20260619 14:42:42.585613 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:42.586887 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:42.588083 1475 hybrid_clock.cc:648] HybridClock initialized: now 1781880162588054 us; error 44 us; skew 500 ppm
Jun 19 14:42:42 dist-test-slave-cr22 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:42.595652 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:42:42.596740 1475 webserver.cc:460] Webserver started at http://127.0.98.62:44241/ using document root <none> and password file <none>
I20260619 14:42:42.597330 1475 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:42.597401 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:42.597625 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:42.599810 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "d9fa752c94804242a7667304400c19b5"
format_stamp: "Formatted at 2026-06-19 14:42:42 on dist-test-slave-cr22"
server_key: "24c4be9ea6389597c511175a7e18f5d2"
server_key_iv: "15f5e62f5a60549c9d146b9294d0e3ae"
server_key_version: "encryptionkey@0"
I20260619 14:42:42.600379 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "d9fa752c94804242a7667304400c19b5"
format_stamp: "Formatted at 2026-06-19 14:42:42 on dist-test-slave-cr22"
server_key: "24c4be9ea6389597c511175a7e18f5d2"
server_key_iv: "15f5e62f5a60549c9d146b9294d0e3ae"
server_key_version: "encryptionkey@0"
I20260619 14:42:42.604279 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:42:42.606706 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:42.608630 1475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:42:42.608831 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "d9fa752c94804242a7667304400c19b5"
format_stamp: "Formatted at 2026-06-19 14:42:42 on dist-test-slave-cr22"
server_key: "24c4be9ea6389597c511175a7e18f5d2"
server_key_iv: "15f5e62f5a60549c9d146b9294d0e3ae"
server_key_version: "encryptionkey@0"
I20260619 14:42:42.608943 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:42.644129 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:42.647363 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:42.647596 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:42.657568 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41483
I20260619 14:42:42.657599 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41483 every 8 connection(s)
I20260619 14:42:42.658555 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:42:42.662299 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:42.667084 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 1475
I20260619 14:42:42.667200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:42:42.667503 392 external_mini_cluster.cc:1428] Setting key 0eee94b48c12bfbdef3b3d705432dff8
I20260619 14:42:42.668321 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5: Bootstrap starting.
I20260619 14:42:42.670765 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:42.671883 1553 log.cc:824] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:42:42.674300 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5: No bootstrap required, opened a new log
Jun 19 14:42:42 dist-test-slave-cr22 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:42.679513 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fa752c94804242a7667304400c19b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41483 } }
I20260619 14:42:42.679823 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:42.679904 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9fa752c94804242a7667304400c19b5, State: Initialized, Role: FOLLOWER
I20260619 14:42:42.680451 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fa752c94804242a7667304400c19b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41483 } }
I20260619 14:42:42.680624 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:42:42.680712 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:42:42.680857 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:42.681933 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fa752c94804242a7667304400c19b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41483 } }
I20260619 14:42:42.682811 1553 leader_election.cc:302] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9fa752c94804242a7667304400c19b5; no voters:
I20260619 14:42:42.683434 1553 leader_election.cc:288] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:42:42.683753 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:42:42.684031 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5 [term 1 LEADER]: Becoming Leader. State: Replica: d9fa752c94804242a7667304400c19b5, State: Running, Role: LEADER
I20260619 14:42:42.684512 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fa752c94804242a7667304400c19b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41483 } }
I20260619 14:42:42.685258 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:42.669069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39626 (local address 127.0.98.62:41483)
0619 14:42:42.669552 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:42.669568 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:42.669604 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:42:42.670302 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:42.670357 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:42.670693 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:42.671057 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:42.672088 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:42.673477 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:42.674736 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:42.675280 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:42.675429 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:42.678698 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:42.678704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:42.678720 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:42.678750 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:42.681186 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:42.681686 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:42.681699 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:42.681704 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:42.681832 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:42.682137 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:42.682141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:42.682142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:42.682888 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:42.683175 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:42.683544 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 14:42:42.684294 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":161,"threads_started":1}
I20260619 14:42:42.685279 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:42:42.685951 1558 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9fa752c94804242a7667304400c19b5. Latest consensus state: current_term: 1 leader_uuid: "d9fa752c94804242a7667304400c19b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fa752c94804242a7667304400c19b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41483 } } }
I20260619 14:42:42.686074 1558 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:42:42.686380 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9fa752c94804242a7667304400c19b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9fa752c94804242a7667304400c19b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41483 } } }
I20260619 14:42:42.686472 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:42:42.687649 1564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:42:42.690917 1564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:42:42.696975 1564 catalog_manager.cc:1347] Generated new cluster ID: 9d391d936cb14c3aa556d7e026b86bec
I20260619 14:42:42.697062 1564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:42:42.711782 1564 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:42:42.713083 1564 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:42:42.718958 1564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5: Generated new TSK 0
I20260619 14:42:42.719797 1564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:42:42.754696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41483
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:42.874559 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:42.874835 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:42.874938 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:42.879571 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:42.879631 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:42.879673 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:42:42.879781 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:42:42.886236 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:42.887388 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:42.888535 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:42.898603 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:42.898648 1579 server_base.cc:1034] running on GCE node
W20260619 14:42:42.898788 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:42.898664 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:42.899480 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:42.900249 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:42.901437 1579 hybrid_clock.cc:648] HybridClock initialized: now 1781880162901415 us; error 33 us; skew 500 ppm
Jun 19 14:42:42 dist-test-slave-cr22 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:42.904376 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:42:42.905200 1579 webserver.cc:460] Webserver started at http://127.0.98.1:33717/ using document root <none> and password file <none>
I20260619 14:42:42.905740 1579 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:42.905800 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:42.905969 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:42.908197 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "4d84caf110594070a00f6eb05caec487"
format_stamp: "Formatted at 2026-06-19 14:42:42 on dist-test-slave-cr22"
server_key: "cfaff08b943e7731c972911a1d817464"
server_key_iv: "aba96e7f6e4dcb205b4ac11d7bf8f0ba"
server_key_version: "encryptionkey@0"
I20260619 14:42:42.908753 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4d84caf110594070a00f6eb05caec487"
format_stamp: "Formatted at 2026-06-19 14:42:42 on dist-test-slave-cr22"
server_key: "cfaff08b943e7731c972911a1d817464"
server_key_iv: "aba96e7f6e4dcb205b4ac11d7bf8f0ba"
server_key_version: "encryptionkey@0"
I20260619 14:42:42.912601 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:42:42.915174 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:42.916666 1579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:42:42.916872 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "4d84caf110594070a00f6eb05caec487"
format_stamp: "Formatted at 2026-06-19 14:42:42 on dist-test-slave-cr22"
server_key: "cfaff08b943e7731c972911a1d817464"
server_key_iv: "aba96e7f6e4dcb205b4ac11d7bf8f0ba"
server_key_version: "encryptionkey@0"
I20260619 14:42:42.916990 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:42.936848 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:42.940164 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:42.940364 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:42.941784 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:42.941833 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:42.941910 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:42.941944 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:42:42 dist-test-slave-cr22 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:42.953547 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43571
I20260619 14:42:42.953641 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43571 every 8 connection(s)
I20260619 14:42:42.954551 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:42:42.957114 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:42.942635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49925 (local address 127.0.98.62:41483)
0619 14:42:42.942801 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:42.942806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:42.943889 (+ 1083us) server_negotiation.cc:407] Connection header received
0619 14:42:42.945200 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:42.945204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:42.945261 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:42.945359 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:42.947051 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:42.947969 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:42.949902 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:42.950254 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:42.950376 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:42.953140 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:42.953145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:42.953149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:42.953179 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:42.954762 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:42.955382 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:42.955385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:42.955386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:42.955518 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:42.955914 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:42.955918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:42.955919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:42.956107 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:42.956196 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:42.956827 (+ 631us) server_negotiation.cc:299] Negotiation successful
0619 14:42:42.956993 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:42:42.957844 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:42.943213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41483 (local address 127.0.98.1:49925)
0619 14:42:42.943699 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:42.943770 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:42.944907 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:42.945561 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:42.945573 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:42.946184 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:42.946862 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:42.946884 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:42.948105 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:42.948108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:42.949738 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:42.949753 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:42.950413 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:42.950417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:42.950609 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:42.951211 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:42.951226 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:42.952918 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:42.954872 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:42.954879 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:42.954894 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:42.955247 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:42.955622 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:42.955625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:42.955627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:42.955776 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:42.956244 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:42.956250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:42.956550 (+ 300us) client_negotiation.cc:769] Sending connection context
0619 14:42:42.956736 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 14:42:42.956987 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":104,"threads_started":1}
I20260619 14:42:42.959170 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41483
I20260619 14:42:42.959311 1710 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:42.959916 1710 heartbeater.cc:511] Master 127.0.98.62:41483 requested a full tablet report, sending...
I20260619 14:42:42.961792 1506 ts_manager.cc:194] Registered new tserver with Master: 4d84caf110594070a00f6eb05caec487 (127.0.98.1:43571)
I20260619 14:42:42.962354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 1579
I20260619 14:42:42.962467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:42:42.962742 392 external_mini_cluster.cc:1428] Setting key e585daa1be145d1be358bb3037ab5e4e
I20260619 14:42:42.963294 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49925
I20260619 14:42:42.971133 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:42.961722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48713 (local address 127.0.98.62:41483)
0619 14:42:42.961881 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:42.961885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:42.962038 (+ 153us) server_negotiation.cc:407] Connection header received
0619 14:42:42.962165 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:42.962168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:42.962208 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:42.962296 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:42.962824 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:42.963560 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:42.964690 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:42.965094 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:42.965214 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:42.966721 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:42.966726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:42.966730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:42.966760 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:42.968647 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:42.969406 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:42.969412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:42.969414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:42.969525 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:42.970009 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:42.970016 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:42.970020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:42.970313 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:42.970407 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:42.970898 (+ 491us) server_negotiation.cc:299] Negotiation successful
0619 14:42:42.971009 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:42:42.971308 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:42.961666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41483 (local address 127.0.98.1:48713)
0619 14:42:42.961935 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:42.961952 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:42.962053 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:42.962341 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:42.962345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:42.962565 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:42.962692 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:42.962700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:42.963718 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:42.963721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:42.964559 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:42.964569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:42.965229 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:42.965232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:42.965321 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:42.965960 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:42.965977 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:42.966562 (+ 585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:42.968977 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:42.968983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:42.968986 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:42.969256 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:42.969658 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:42.969665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:42.969671 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:42.969785 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:42.970513 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:42.970519 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:42.970657 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 14:42:42.970800 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 14:42:42.971018 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:42:42.996487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41483
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:43.120952 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:43.121269 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:43.121364 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:43.125535 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:43.125626 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:43.125705 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:42:43.125787 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:42:43.131170 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:43.132329 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:43.133366 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:43.143304 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:43.143379 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:43.143369 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:43.143721 1719 server_base.cc:1034] running on GCE node
I20260619 14:42:43.144140 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:43.144848 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:43.146050 1719 hybrid_clock.cc:648] HybridClock initialized: now 1781880163146031 us; error 29 us; skew 500 ppm
Jun 19 14:42:43 dist-test-slave-cr22 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:43.149199 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:42:43.150633 1719 webserver.cc:460] Webserver started at http://127.0.98.2:36209/ using document root <none> and password file <none>
I20260619 14:42:43.151393 1719 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:43.151566 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:43.151825 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:43.154215 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "21eb4bf8edc74177a4715450a2caef21"
format_stamp: "Formatted at 2026-06-19 14:42:43 on dist-test-slave-cr22"
server_key: "0bdb698461cbbdcdb4d40cda4f949361"
server_key_iv: "17e96de0bc326ecc47e6e0fff0dcec8e"
server_key_version: "encryptionkey@0"
I20260619 14:42:43.154757 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "21eb4bf8edc74177a4715450a2caef21"
format_stamp: "Formatted at 2026-06-19 14:42:43 on dist-test-slave-cr22"
server_key: "0bdb698461cbbdcdb4d40cda4f949361"
server_key_iv: "17e96de0bc326ecc47e6e0fff0dcec8e"
server_key_version: "encryptionkey@0"
I20260619 14:42:43.159248 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 14:42:43.162321 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:43.163856 1719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:42:43.164005 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "21eb4bf8edc74177a4715450a2caef21"
format_stamp: "Formatted at 2026-06-19 14:42:43 on dist-test-slave-cr22"
server_key: "0bdb698461cbbdcdb4d40cda4f949361"
server_key_iv: "17e96de0bc326ecc47e6e0fff0dcec8e"
server_key_version: "encryptionkey@0"
I20260619 14:42:43.164129 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:43.175840 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:43.179133 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:43.179958 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:43.181841 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:43.181918 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:43.181991 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:43.182042 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:42:43 dist-test-slave-cr22 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:43.194573 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41561
I20260619 14:42:43.194645 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41561 every 8 connection(s)
I20260619 14:42:43.195567 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:42:43.198235 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.182790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38559 (local address 127.0.98.62:41483)
0619 14:42:43.182948 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:43.182952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:43.184378 (+ 1426us) server_negotiation.cc:407] Connection header received
0619 14:42:43.185431 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:43.185435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:43.185497 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:43.185593 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:43.187655 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.188489 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.190153 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.190836 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.190979 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.194149 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:43.194152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:43.194154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:43.194178 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:43.196122 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:43.196677 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:43.196685 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:43.196686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:43.196765 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:43.197200 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:43.197203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:43.197204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:43.197396 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:43.197497 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:43.198014 (+ 517us) server_negotiation.cc:299] Negotiation successful
0619 14:42:43.198118 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 14:42:43.198946 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.183310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41483 (local address 127.0.98.2:38559)
0619 14:42:43.184189 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:43.184249 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:43.185167 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:43.185831 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:43.185841 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:43.186523 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:43.187477 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.187502 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.188630 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.188634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.189983 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.189998 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.191029 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.191033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.191213 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.192009 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:43.192027 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:43.193938 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:43.196259 (+ 2321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:43.196265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:43.196278 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:43.196550 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:43.196930 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:43.196932 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:43.196934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:43.197065 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:43.197522 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:43.197535 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:43.197760 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 14:42:43.197914 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 14:42:43.198152 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":175,"threads_started":1}
I20260619 14:42:43.200213 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41483
I20260619 14:42:43.200555 1850 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:43.201229 1850 heartbeater.cc:511] Master 127.0.98.62:41483 requested a full tablet report, sending...
I20260619 14:42:43.202276 1506 ts_manager.cc:194] Registered new tserver with Master: 21eb4bf8edc74177a4715450a2caef21 (127.0.98.2:41561)
I20260619 14:42:43.202831 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38559
I20260619 14:42:43.203336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 1719
I20260619 14:42:43.203482 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:42:43.203765 392 external_mini_cluster.cc:1428] Setting key 21f143ae4be197e79efe26f065beb94b
I20260619 14:42:43.209964 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.202228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39657 (local address 127.0.98.62:41483)
0619 14:42:43.202387 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:43.202392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:43.202620 (+ 228us) server_negotiation.cc:407] Connection header received
0619 14:42:43.202847 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:43.202850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:43.202896 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:43.203003 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:43.203554 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.204048 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.205175 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.205461 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.205580 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.206652 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:43.206656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:43.206659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:43.206681 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:43.208240 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:43.208730 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:43.208733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:43.208735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:43.208815 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:43.209108 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:43.209111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:43.209113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:43.209320 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:43.209429 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:43.209700 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 14:42:43.209809 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:42:43.209976 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.202157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41483 (local address 127.0.98.2:39657)
0619 14:42:43.202530 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:43.202543 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:43.202669 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:43.203038 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:43.203041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:43.203307 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:43.203428 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.203460 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.204214 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.204220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.205049 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.205059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.205580 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.205583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.205671 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.206195 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:43.206208 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:43.206510 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:43.208385 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:43.208389 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:43.208391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:43.208607 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:43.208919 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:43.208922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:43.208924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:43.209011 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:43.209429 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:43.209433 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:43.209528 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 14:42:43.209635 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:42:43.209794 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:42:43.235287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41483
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:43.350586 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:43.350920 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:43.351006 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:43.355226 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:43.355383 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:43.355489 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:42:43.355561 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:42:43.362416 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:43.364120 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:43.365958 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:43.378244 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:43.378240 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:43.378332 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:43.378312 1859 server_base.cc:1034] running on GCE node
I20260619 14:42:43.379068 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:43.380082 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:43.381335 1859 hybrid_clock.cc:648] HybridClock initialized: now 1781880163381326 us; error 32 us; skew 500 ppm
Jun 19 14:42:43 dist-test-slave-cr22 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:43.384296 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:42:43.385324 1859 webserver.cc:460] Webserver started at http://127.0.98.3:45097/ using document root <none> and password file <none>
I20260619 14:42:43.385995 1859 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:43.386096 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:43.386381 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:43.388859 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "2e60821de1914d60bd786b394290d9b9"
format_stamp: "Formatted at 2026-06-19 14:42:43 on dist-test-slave-cr22"
server_key: "ed27b2734921cbd168ac4d69b2c3731c"
server_key_iv: "620150c306afe50c0b1fa28602b6f30e"
server_key_version: "encryptionkey@0"
I20260619 14:42:43.389472 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2e60821de1914d60bd786b394290d9b9"
format_stamp: "Formatted at 2026-06-19 14:42:43 on dist-test-slave-cr22"
server_key: "ed27b2734921cbd168ac4d69b2c3731c"
server_key_iv: "620150c306afe50c0b1fa28602b6f30e"
server_key_version: "encryptionkey@0"
I20260619 14:42:43.393690 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:42:43.396526 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:43.397955 1859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:42:43.398097 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "2e60821de1914d60bd786b394290d9b9"
format_stamp: "Formatted at 2026-06-19 14:42:43 on dist-test-slave-cr22"
server_key: "ed27b2734921cbd168ac4d69b2c3731c"
server_key_iv: "620150c306afe50c0b1fa28602b6f30e"
server_key_version: "encryptionkey@0"
I20260619 14:42:43.398221 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:43.411089 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:43.414395 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:43.414628 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:43.416287 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:43.416338 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:43.416404 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:43.416438 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:42:43 dist-test-slave-cr22 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:43.428483 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43901
I20260619 14:42:43.428532 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43901 every 8 connection(s)
I20260619 14:42:43.429594 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:42:43.431375 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.417243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41483 (local address 127.0.98.3:56181)
0619 14:42:43.417676 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:43.417725 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:43.418643 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:43.419487 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:43.419495 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:43.420110 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:43.420705 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.420737 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.422088 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.422093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.422974 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.422983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.423734 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.423738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.423982 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.424621 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:43.424650 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:43.426550 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:43.428590 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:43.428596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:43.428610 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:43.428889 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:43.429181 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:43.429183 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:43.429185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:43.429301 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:43.429798 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:43.429804 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:43.430114 (+ 310us) client_negotiation.cc:769] Sending connection context
0619 14:42:43.430335 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 14:42:43.430657 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":128,"threads_started":1}
I20260619 14:42:43.431856 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.416767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56181 (local address 127.0.98.62:41483)
0619 14:42:43.416920 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:43.416925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:43.417840 (+ 915us) server_negotiation.cc:407] Connection header received
0619 14:42:43.418903 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:43.418907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:43.418966 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:43.419028 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:43.420869 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.421914 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.423109 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.423555 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.423696 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.426719 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:43.426723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:43.426726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:43.426753 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:43.428478 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:43.428993 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:43.428997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:43.428998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:43.429091 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:43.429400 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:43.429403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:43.429405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:43.429650 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:43.429746 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:43.431592 (+ 1846us) server_negotiation.cc:299] Negotiation successful
0619 14:42:43.431717 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:42:43.432529 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 1859
I20260619 14:42:43.432669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:42:43.432904 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41483
I20260619 14:42:43.432961 392 external_mini_cluster.cc:1428] Setting key c70d9859630be1fb4286674398e95936
I20260619 14:42:43.433040 1990 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:43.433658 1990 heartbeater.cc:511] Master 127.0.98.62:41483 requested a full tablet report, sending...
I20260619 14:42:43.434765 1506 ts_manager.cc:194] Registered new tserver with Master: 2e60821de1914d60bd786b394290d9b9 (127.0.98.3:43901)
I20260619 14:42:43.435566 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56181
I20260619 14:42:43.442219 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.435111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41483 (local address 127.0.98.3:60835)
0619 14:42:43.435386 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:43.435401 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:43.435509 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:43.435772 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:43.435775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:43.435989 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:43.436096 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.436102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.436847 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.436850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.437406 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.437414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.437860 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.437864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.437953 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.438486 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:43.438500 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:43.438826 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:43.440600 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:43.440606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:43.440608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:43.440893 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:43.441188 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:43.441191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:43.441193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:43.441288 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:43.441668 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:43.441671 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:43.441763 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 14:42:43.441873 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 14:42:43.442013 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":85,"threads_started":1}
I20260619 14:42:43.442214 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.435458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60835 (local address 127.0.98.62:41483)
0619 14:42:43.435580 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:43.435585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:43.435597 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:42:43.435627 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:43.435629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:43.435673 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:43.435756 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:43.436211 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.436725 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.437525 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.437751 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.437859 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.438965 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:43.438968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:43.438971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:43.438994 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:43.440494 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:43.440995 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:43.441000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:43.441002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:43.441080 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:43.441386 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:43.441389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:43.441390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:43.441561 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:43.441670 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:43.441936 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 14:42:43.442043 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:42:43.447046 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:42:43 dist-test-slave-cr22 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880163, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:42:43 dist-test-slave-cr22 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880163, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:43.474303 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.465176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.0.98.62:41483)
0619 14:42:43.465330 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:43.465334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:43.465346 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:42:43.465387 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:43.465390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:43.465427 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:43.465505 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:43.466286 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.466844 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.467970 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.468181 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.468268 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.470931 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:43.470934 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:43.470936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:43.470961 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:43.472605 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:43.473030 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:43.473033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:43.473034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:43.473117 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:43.473462 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:43.473465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:43.473466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:43.473651 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:43.473757 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:43.474073 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 14:42:43.474170 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 14:42:43.477885 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:42:43.481622 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:42:43.496747 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.490712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.0.98.2:41561)
0619 14:42:43.491116 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:43.491129 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:43.491634 (+ 505us) server_negotiation.cc:407] Connection header received
0619 14:42:43.491729 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:43.491737 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:43.491963 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:43.492114 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:43.492416 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.493671 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.495152 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.496000 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.496117 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.496302 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:43.496471 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 14:42:43.496568 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":109,"threads_started":1}
I20260619 14:42:43.496747 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.490510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41561 (local address 127.0.0.1:43414)
0619 14:42:43.491285 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:43.491331 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:43.491494 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:43.492100 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:43.492106 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:43.492127 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:43.492287 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.492298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.493790 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.493794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.495017 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.495028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.496129 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.496133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.496257 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.496314 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 14:42:43.496421 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:42:43.496553 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":8,"spinlock_wait_cycles":3328,"thread_start_us":72,"threads_started":1}
I20260619 14:42:43.497049 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.491275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.0.98.1:43571)
0619 14:42:43.491719 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:43.491725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:43.492431 (+ 706us) server_negotiation.cc:407] Connection header received
0619 14:42:43.492547 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:43.492573 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:43.492737 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:43.492864 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:43.493156 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.493952 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.495368 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.496137 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.496228 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.496345 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:43.496817 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 14:42:43.496932 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":87,"threads_started":1}
I20260619 14:42:43.497089 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.491282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37708 (local address 127.0.98.3:43901)
0619 14:42:43.491798 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:43.491809 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:43.491833 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:42:43.491913 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:43.491917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:43.492095 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:43.492605 (+ 510us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:43.492842 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.493871 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.495481 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.496191 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.496652 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.496798 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:43.496851 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 14:42:43.496944 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":96,"threads_started":1}
I20260619 14:42:43.497130 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.490705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43901 (local address 127.0.0.1:37708)
0619 14:42:43.491286 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:43.491332 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:43.491493 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:43.492599 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:43.492602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:43.492620 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:43.492715 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.492723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.493980 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.493986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.495353 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.495362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.496321 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.496324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.496443 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.496468 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:42:43.496510 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 14:42:43.496560 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":154,"spinlock_wait_cycles":2432,"thread_start_us":112,"threads_started":1}
I20260619 14:42:43.497428 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.490897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43571 (local address 127.0.0.1:45470)
0619 14:42:43.492317 (+ 1420us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:43.492331 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:43.492444 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:43.492975 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:43.492977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:43.492990 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:43.493049 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.493052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.494051 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.494054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.495247 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.495256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.496406 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.496412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.496531 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.496555 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:42:43.496643 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 14:42:43.496727 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1363,"mutex_wait_us":60,"thread_start_us":96,"threads_started":1}
I20260619 14:42:43.499945 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 042fd8206c0e4a5baf065c59d844488b (DEFAULT_TABLE table=test-table [id=bf4f5067c5b94ed4a5f258c9f829e8b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:43.499941 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 042fd8206c0e4a5baf065c59d844488b (DEFAULT_TABLE table=test-table [id=bf4f5067c5b94ed4a5f258c9f829e8b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:43.500914 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 042fd8206c0e4a5baf065c59d844488b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:43.500923 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 042fd8206c0e4a5baf065c59d844488b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:43.504104 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 042fd8206c0e4a5baf065c59d844488b (DEFAULT_TABLE table=test-table [id=bf4f5067c5b94ed4a5f258c9f829e8b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:43.504957 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 042fd8206c0e4a5baf065c59d844488b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:43.507890 2013 tablet_bootstrap.cc:492] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9: Bootstrap starting.
I20260619 14:42:43.509953 2013 tablet_bootstrap.cc:654] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:43.510696 2013 log.cc:824] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:42:43.512722 2013 tablet_bootstrap.cc:492] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9: No bootstrap required, opened a new log
I20260619 14:42:43.512908 2013 ts_tablet_manager.cc:1397] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:42:43.512959 2016 tablet_bootstrap.cc:492] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21: Bootstrap starting.
I20260619 14:42:43.513463 2015 tablet_bootstrap.cc:492] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487: Bootstrap starting.
I20260619 14:42:43.515110 2015 tablet_bootstrap.cc:654] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:43.515244 2016 tablet_bootstrap.cc:654] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:43.515924 2015 log.cc:824] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487: Log is configured to *not* fsync() on all Append() calls
I20260619 14:42:43.516070 2016 log.cc:824] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21: Log is configured to *not* fsync() on all Append() calls
I20260619 14:42:43.516922 2013 raft_consensus.cc:348] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } } peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } }
I20260619 14:42:43.517187 2013 raft_consensus.cc:374] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:43.517275 2013 raft_consensus.cc:729] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e60821de1914d60bd786b394290d9b9, State: Initialized, Role: FOLLOWER
I20260619 14:42:43.517740 2013 consensus_queue.cc:260] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } } peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } }
I20260619 14:42:43.517915 2015 tablet_bootstrap.cc:492] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487: No bootstrap required, opened a new log
I20260619 14:42:43.517925 2016 tablet_bootstrap.cc:492] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21: No bootstrap required, opened a new log
I20260619 14:42:43.518097 2015 ts_tablet_manager.cc:1397] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260619 14:42:43.518118 2016 ts_tablet_manager.cc:1397] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:42:43.518407 1990 heartbeater.cc:503] Master 127.0.98.62:41483 was elected leader, sending a full tablet report...
I20260619 14:42:43.518744 2013 ts_tablet_manager.cc:1428] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:42:43.522781 2016 raft_consensus.cc:348] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } } peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } }
I20260619 14:42:43.522781 2015 raft_consensus.cc:348] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } } peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } }
I20260619 14:42:43.523053 2016 raft_consensus.cc:374] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:43.523053 2015 raft_consensus.cc:374] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:43.523119 2016 raft_consensus.cc:729] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21eb4bf8edc74177a4715450a2caef21, State: Initialized, Role: FOLLOWER
I20260619 14:42:43.523119 2015 raft_consensus.cc:729] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d84caf110594070a00f6eb05caec487, State: Initialized, Role: FOLLOWER
I20260619 14:42:43.523590 2015 consensus_queue.cc:260] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } } peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } }
I20260619 14:42:43.523590 2016 consensus_queue.cc:260] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } } peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } }
I20260619 14:42:43.524245 1850 heartbeater.cc:503] Master 127.0.98.62:41483 was elected leader, sending a full tablet report...
I20260619 14:42:43.524243 1710 heartbeater.cc:503] Master 127.0.98.62:41483 was elected leader, sending a full tablet report...
I20260619 14:42:43.524585 2015 ts_tablet_manager.cc:1428] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:42:43.525537 2016 ts_tablet_manager.cc:1428] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260619 14:42:43.680594 1991 tablet.cc:2369] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:42:43.696827 1851 tablet.cc:2369] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:42:43.705896 1711 tablet.cc:2369] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:42:43.728516 2019 raft_consensus.cc:482] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:42:43.728709 2019 raft_consensus.cc:504] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } } peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } }
I20260619 14:42:43.729883 2019 leader_election.cc:288] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21eb4bf8edc74177a4715450a2caef21 (127.0.98.2:41561), 4d84caf110594070a00f6eb05caec487 (127.0.98.1:43571)
I20260619 14:42:43.734438 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.730108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41561 (local address 127.0.98.3:39233)
0619 14:42:43.730296 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:43.730315 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:43.730422 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:43.730776 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:43.730780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:43.730799 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:43.731059 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.731069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.731984 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.731987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.733168 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.733178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.734015 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.734018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.734151 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.734172 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:42:43.734246 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:42:43.734306 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":18}
I20260619 14:42:43.734568 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.730185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39233 (local address 127.0.98.2:41561)
0619 14:42:43.730306 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:43.730311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:43.730395 (+ 84us) server_negotiation.cc:407] Connection header received
0619 14:42:43.730570 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:43.730576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:43.730655 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:43.730738 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:43.731212 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.731848 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.733340 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.733871 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.734005 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.734085 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:43.734345 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 14:42:43.734432 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:42:43.735213 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.730166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43571 (local address 127.0.98.3:50493)
0619 14:42:43.730579 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:43.730590 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:43.730683 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:43.731021 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:43.731023 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:43.731077 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:43.731684 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.731693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.732627 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.732630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.733701 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.733709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.734823 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.734827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.734941 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.734970 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:42:43.735058 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 14:42:43.735108 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":1,"thread_start_us":146,"threads_started":1}
I20260619 14:42:43.735345 1796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "042fd8206c0e4a5baf065c59d844488b" candidate_uuid: "2e60821de1914d60bd786b394290d9b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eb4bf8edc74177a4715450a2caef21" is_pre_election: true
I20260619 14:42:43.735671 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.730403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50493 (local address 127.0.98.1:43571)
0619 14:42:43.730557 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:43.730561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:43.730664 (+ 103us) server_negotiation.cc:407] Connection header received
0619 14:42:43.730808 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:43.730811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:43.730861 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:43.730928 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:43.731853 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.732514 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.733908 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.734618 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.734756 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.734840 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:43.735407 (+ 567us) server_negotiation.cc:299] Negotiation successful
0619 14:42:43.735559 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:42:43.735726 1796 raft_consensus.cc:2454] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e60821de1914d60bd786b394290d9b9 in term 0.
I20260619 14:42:43.736285 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "042fd8206c0e4a5baf065c59d844488b" candidate_uuid: "2e60821de1914d60bd786b394290d9b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d84caf110594070a00f6eb05caec487" is_pre_election: true
I20260619 14:42:43.736472 1877 leader_election.cc:302] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eb4bf8edc74177a4715450a2caef21, 2e60821de1914d60bd786b394290d9b9; no voters:
I20260619 14:42:43.736567 1663 raft_consensus.cc:2454] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e60821de1914d60bd786b394290d9b9 in term 0.
I20260619 14:42:43.736795 2019 raft_consensus.cc:2781] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:42:43.736897 2019 raft_consensus.cc:482] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:42:43.736958 2019 raft_consensus.cc:3037] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:43.738106 2019 raft_consensus.cc:504] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } } peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } }
I20260619 14:42:43.738525 2019 leader_election.cc:288] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [CANDIDATE]: Term 1 election: Requested vote from peers 21eb4bf8edc74177a4715450a2caef21 (127.0.98.2:41561), 4d84caf110594070a00f6eb05caec487 (127.0.98.1:43571)
I20260619 14:42:43.738878 1796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "042fd8206c0e4a5baf065c59d844488b" candidate_uuid: "2e60821de1914d60bd786b394290d9b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eb4bf8edc74177a4715450a2caef21"
I20260619 14:42:43.738947 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "042fd8206c0e4a5baf065c59d844488b" candidate_uuid: "2e60821de1914d60bd786b394290d9b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d84caf110594070a00f6eb05caec487"
I20260619 14:42:43.739019 1796 raft_consensus.cc:3037] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:43.739097 1663 raft_consensus.cc:3037] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:43.740191 1796 raft_consensus.cc:2454] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e60821de1914d60bd786b394290d9b9 in term 1.
I20260619 14:42:43.740403 1663 raft_consensus.cc:2454] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e60821de1914d60bd786b394290d9b9 in term 1.
I20260619 14:42:43.740581 1877 leader_election.cc:302] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eb4bf8edc74177a4715450a2caef21, 2e60821de1914d60bd786b394290d9b9; no voters:
I20260619 14:42:43.740800 2019 raft_consensus.cc:2781] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:42:43.741184 2019 raft_consensus.cc:686] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [term 1 LEADER]: Becoming Leader. State: Replica: 2e60821de1914d60bd786b394290d9b9, State: Running, Role: LEADER
I20260619 14:42:43.741528 2019 consensus_queue.cc:237] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } } peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } }
I20260619 14:42:43.744982 1506 catalog_manager.cc:5495] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e60821de1914d60bd786b394290d9b9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2e60821de1914d60bd786b394290d9b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } health_report { overall_health: UNKNOWN } } }
I20260619 14:42:43.796870 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.792972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.0.98.3:43901)
0619 14:42:43.793124 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:43.793127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:43.793143 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:42:43.793306 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:43.793308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:43.793365 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:43.793455 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:42:43.793711 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.794223 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.795143 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.795403 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.795565 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.795885 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:42:43.796335 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:42:43.796424 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:43.796639 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 14:42:43.796694 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:42:43.803170 1663 raft_consensus.cc:1261] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487 [term 1 FOLLOWER]: Refusing update from remote peer 2e60821de1914d60bd786b394290d9b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:42:43.803187 1796 raft_consensus.cc:1261] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21 [term 1 FOLLOWER]: Refusing update from remote peer 2e60821de1914d60bd786b394290d9b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:42:43.803961 2023 consensus_queue.cc:1035] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:42:43.804183 2019 consensus_queue.cc:1035] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:42:43.814605 2028 mvcc.cc:205] Tried to move back new op lower bound from 7298581150930444288 to 7298581150688026624. Current Snapshot: MvccSnapshot[applied={T|T < 7298581150930444288}]
I20260619 14:42:43.824709 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:42:43.824872 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:42:43.827266 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5: Sending DeleteTablet for 3 replicas of tablet 042fd8206c0e4a5baf065c59d844488b
I20260619 14:42:43.828141 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 042fd8206c0e4a5baf065c59d844488b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:42:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43414
I20260619 14:42:43.828261 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 042fd8206c0e4a5baf065c59d844488b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:42:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45470
I20260619 14:42:43.828788 2036 tablet_replica.cc:330] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21: stopping tablet replica
I20260619 14:42:43.828790 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 042fd8206c0e4a5baf065c59d844488b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:42:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37708
I20260619 14:42:43.828977 2037 tablet_replica.cc:330] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487: stopping tablet replica
I20260619 14:42:43.828959 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 14:42:43.829176 2036 raft_consensus.cc:2229] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:42:43.829300 2037 raft_consensus.cc:2229] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 14:42:43.829368 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:42:43.829558 2036 raft_consensus.cc:2258] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:42:43.829627 2037 raft_consensus.cc:2258] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:42:43.830185 2038 tablet_replica.cc:330] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9: stopping tablet replica
I20260619 14:42:43.830551 2038 raft_consensus.cc:2229] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:42:43.831005 2038 raft_consensus.cc:2258] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:42:43.831190 2037 ts_tablet_manager.cc:1905] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:42:43.832671 2038 ts_tablet_manager.cc:1905] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:42:43.833781 2036 ts_tablet_manager.cc:1905] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:42:43.834204 2037 ts_tablet_manager.cc:1918] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:42:43.834345 2037 log.cc:1196] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-0/wal/wals/042fd8206c0e4a5baf065c59d844488b
I20260619 14:42:43.834939 2037 ts_tablet_manager.cc:1939] T 042fd8206c0e4a5baf065c59d844488b P 4d84caf110594070a00f6eb05caec487: Deleting consensus metadata
I20260619 14:42:43.835023 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 3e9a10e7ce1d4605a2f79a8ac774d7c6 (DEFAULT_TABLE table=test-table [id=af6591052ce74d3baa5eb9e5985882a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:43.835284 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e9a10e7ce1d4605a2f79a8ac774d7c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:43.835290 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 3e9a10e7ce1d4605a2f79a8ac774d7c6 (DEFAULT_TABLE table=test-table [id=af6591052ce74d3baa5eb9e5985882a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:43.835549 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e9a10e7ce1d4605a2f79a8ac774d7c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:43.837340 2036 ts_tablet_manager.cc:1918] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:42:43.837520 2015 tablet_bootstrap.cc:492] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487: Bootstrap starting.
I20260619 14:42:43.837759 1494 catalog_manager.cc:4841] TS 4d84caf110594070a00f6eb05caec487 (127.0.98.1:43571): tablet 042fd8206c0e4a5baf065c59d844488b (table test-table [id=bf4f5067c5b94ed4a5f258c9f829e8b5]) successfully deleted
I20260619 14:42:43.837849 2016 tablet_bootstrap.cc:492] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 21eb4bf8edc74177a4715450a2caef21: Bootstrap starting.
I20260619 14:42:43.838476 2015 tablet_bootstrap.cc:654] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:43.838738 2016 tablet_bootstrap.cc:654] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 21eb4bf8edc74177a4715450a2caef21: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:43.839205 2036 log.cc:1196] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/042fd8206c0e4a5baf065c59d844488b
I20260619 14:42:43.839751 2038 ts_tablet_manager.cc:1918] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:42:43.839807 2036 ts_tablet_manager.cc:1939] T 042fd8206c0e4a5baf065c59d844488b P 21eb4bf8edc74177a4715450a2caef21: Deleting consensus metadata
I20260619 14:42:43.839874 2038 log.cc:1196] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/042fd8206c0e4a5baf065c59d844488b
I20260619 14:42:43.840186 2038 ts_tablet_manager.cc:1939] T 042fd8206c0e4a5baf065c59d844488b P 2e60821de1914d60bd786b394290d9b9: Deleting consensus metadata
I20260619 14:42:43.840459 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 3e9a10e7ce1d4605a2f79a8ac774d7c6 (DEFAULT_TABLE table=test-table [id=af6591052ce74d3baa5eb9e5985882a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:43.840691 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e9a10e7ce1d4605a2f79a8ac774d7c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:43.841354 1492 catalog_manager.cc:4841] TS 2e60821de1914d60bd786b394290d9b9 (127.0.98.3:43901): tablet 042fd8206c0e4a5baf065c59d844488b (table test-table [id=bf4f5067c5b94ed4a5f258c9f829e8b5]) successfully deleted
I20260619 14:42:43.842056 2016 tablet_bootstrap.cc:492] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 21eb4bf8edc74177a4715450a2caef21: No bootstrap required, opened a new log
I20260619 14:42:43.842116 2016 ts_tablet_manager.cc:1397] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 21eb4bf8edc74177a4715450a2caef21: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260619 14:42:43.842487 2016 raft_consensus.cc:348] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 21eb4bf8edc74177a4715450a2caef21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } }
I20260619 14:42:43.842568 2016 raft_consensus.cc:374] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 21eb4bf8edc74177a4715450a2caef21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:43.842592 2016 raft_consensus.cc:729] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 21eb4bf8edc74177a4715450a2caef21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21eb4bf8edc74177a4715450a2caef21, State: Initialized, Role: FOLLOWER
I20260619 14:42:43.842734 2016 consensus_queue.cc:260] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 21eb4bf8edc74177a4715450a2caef21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } }
I20260619 14:42:43.842933 2013 tablet_bootstrap.cc:492] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 2e60821de1914d60bd786b394290d9b9: Bootstrap starting.
I20260619 14:42:43.843006 2016 ts_tablet_manager.cc:1428] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 21eb4bf8edc74177a4715450a2caef21: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 14:42:43.843869 2013 tablet_bootstrap.cc:654] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 2e60821de1914d60bd786b394290d9b9: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:43.844588 2015 tablet_bootstrap.cc:492] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487: No bootstrap required, opened a new log
I20260619 14:42:43.844688 2015 ts_tablet_manager.cc:1397] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20260619 14:42:43.844986 2013 tablet_bootstrap.cc:492] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 2e60821de1914d60bd786b394290d9b9: No bootstrap required, opened a new log
I20260619 14:42:43.845054 2013 ts_tablet_manager.cc:1397] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 2e60821de1914d60bd786b394290d9b9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:42:43.845357 2015 raft_consensus.cc:348] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } }
I20260619 14:42:43.845441 2015 raft_consensus.cc:374] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:43.845461 2015 raft_consensus.cc:729] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d84caf110594070a00f6eb05caec487, State: Initialized, Role: FOLLOWER
I20260619 14:42:43.845474 2013 raft_consensus.cc:348] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 2e60821de1914d60bd786b394290d9b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } }
I20260619 14:42:43.845570 2013 raft_consensus.cc:374] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 2e60821de1914d60bd786b394290d9b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:43.845599 2013 raft_consensus.cc:729] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 2e60821de1914d60bd786b394290d9b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e60821de1914d60bd786b394290d9b9, State: Initialized, Role: FOLLOWER
I20260619 14:42:43.845610 2015 consensus_queue.cc:260] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } }
I20260619 14:42:43.845718 2013 consensus_queue.cc:260] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 2e60821de1914d60bd786b394290d9b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } }
I20260619 14:42:43.845894 2015 ts_tablet_manager.cc:1428] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:42:43.845963 2013 ts_tablet_manager.cc:1428] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 2e60821de1914d60bd786b394290d9b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:42:43.847472 1493 catalog_manager.cc:4841] TS 21eb4bf8edc74177a4715450a2caef21 (127.0.98.2:41561): tablet 042fd8206c0e4a5baf065c59d844488b (table test-table [id=bf4f5067c5b94ed4a5f258c9f829e8b5]) successfully deleted
I20260619 14:42:43.942286 2020 raft_consensus.cc:482] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:42:43.942441 2020 raft_consensus.cc:504] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } }
I20260619 14:42:43.943715 2020 leader_election.cc:288] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21eb4bf8edc74177a4715450a2caef21 (127.0.98.2:41561), 2e60821de1914d60bd786b394290d9b9 (127.0.98.3:43901)
I20260619 14:42:43.948230 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.944215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49147 (local address 127.0.98.3:43901)
0619 14:42:43.944524 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:43.944528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:43.944543 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:42:43.944597 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:43.944600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:43.944677 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:43.944763 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:43.945053 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.945730 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.946966 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.947593 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.947701 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.947770 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:43.948021 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 14:42:43.948094 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260619 14:42:43.948233 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.943956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43901 (local address 127.0.98.1:49147)
0619 14:42:43.944308 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:43.944325 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:43.944442 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:43.944794 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:43.944798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:43.944811 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:43.944955 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.944964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.945843 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.945848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.946830 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.946840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.947756 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.947760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.947873 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.947898 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:42:43.947979 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:42:43.948047 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":78,"threads_started":1}
I20260619 14:42:43.948848 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.944076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41923 (local address 127.0.98.2:41561)
0619 14:42:43.944409 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:43.944415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:43.944705 (+ 290us) server_negotiation.cc:407] Connection header received
0619 14:42:43.945028 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:43.945032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:43.945081 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:43.945177 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:43.945648 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.946370 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.947475 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.948041 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.948133 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.948186 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:43.948655 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 14:42:43.948721 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246}
I20260619 14:42:43.948860 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.943956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41561 (local address 127.0.98.1:41923)
0619 14:42:43.944607 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:43.944618 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:43.944794 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:43.945195 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:43.945198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:43.945217 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:43.945377 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.945384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.946495 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.946499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.947339 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.947346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.948242 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.948248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.948520 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.948550 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:42:43.948638 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 14:42:43.948701 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":123,"thread_start_us":110,"threads_started":1}
I20260619 14:42:43.948943 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e9a10e7ce1d4605a2f79a8ac774d7c6" candidate_uuid: "4d84caf110594070a00f6eb05caec487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e60821de1914d60bd786b394290d9b9" is_pre_election: true
I20260619 14:42:43.949190 1944 raft_consensus.cc:2454] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 2e60821de1914d60bd786b394290d9b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d84caf110594070a00f6eb05caec487 in term 0.
I20260619 14:42:43.949312 1796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e9a10e7ce1d4605a2f79a8ac774d7c6" candidate_uuid: "4d84caf110594070a00f6eb05caec487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eb4bf8edc74177a4715450a2caef21" is_pre_election: true
I20260619 14:42:43.949469 1796 raft_consensus.cc:2454] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 21eb4bf8edc74177a4715450a2caef21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d84caf110594070a00f6eb05caec487 in term 0.
I20260619 14:42:43.949687 1596 leader_election.cc:302] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e60821de1914d60bd786b394290d9b9, 4d84caf110594070a00f6eb05caec487; no voters:
I20260619 14:42:43.949999 2020 raft_consensus.cc:2781] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:42:43.950068 2020 raft_consensus.cc:482] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:42:43.950109 2020 raft_consensus.cc:3037] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:43.951411 2020 raft_consensus.cc:504] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } }
I20260619 14:42:43.951840 2020 leader_election.cc:288] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [CANDIDATE]: Term 1 election: Requested vote from peers 21eb4bf8edc74177a4715450a2caef21 (127.0.98.2:41561), 2e60821de1914d60bd786b394290d9b9 (127.0.98.3:43901)
I20260619 14:42:43.952246 1796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e9a10e7ce1d4605a2f79a8ac774d7c6" candidate_uuid: "4d84caf110594070a00f6eb05caec487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eb4bf8edc74177a4715450a2caef21"
I20260619 14:42:43.952383 1796 raft_consensus.cc:3037] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 21eb4bf8edc74177a4715450a2caef21 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:43.952394 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e9a10e7ce1d4605a2f79a8ac774d7c6" candidate_uuid: "4d84caf110594070a00f6eb05caec487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e60821de1914d60bd786b394290d9b9"
I20260619 14:42:43.952503 1944 raft_consensus.cc:3037] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 2e60821de1914d60bd786b394290d9b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:43.953439 1796 raft_consensus.cc:2454] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 21eb4bf8edc74177a4715450a2caef21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d84caf110594070a00f6eb05caec487 in term 1.
I20260619 14:42:43.953703 1944 raft_consensus.cc:2454] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 2e60821de1914d60bd786b394290d9b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d84caf110594070a00f6eb05caec487 in term 1.
I20260619 14:42:43.953979 1596 leader_election.cc:302] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e60821de1914d60bd786b394290d9b9, 4d84caf110594070a00f6eb05caec487; no voters:
I20260619 14:42:43.954241 2020 raft_consensus.cc:2781] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:42:43.954461 2020 raft_consensus.cc:686] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [term 1 LEADER]: Becoming Leader. State: Replica: 4d84caf110594070a00f6eb05caec487, State: Running, Role: LEADER
I20260619 14:42:43.954718 2020 consensus_queue.cc:237] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } }
I20260619 14:42:43.958596 1505 catalog_manager.cc:5495] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d84caf110594070a00f6eb05caec487 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4d84caf110594070a00f6eb05caec487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:42:43 dist-test-slave-cr22 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:43.999692 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.986274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41483 (local address 127.0.0.1:53840)
0619 14:42:43.986771 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:43.986785 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:43.986918 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:43.987468 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:43.987472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:43.988416 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:43.988581 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.988590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.989623 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.989629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.990678 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:43.990701 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.991544 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.991549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:43.991664 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.992628 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:43.992647 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:43.994600 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:43.997725 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:43.997740 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:43.997749 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:43.998066 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:43.998410 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:43.998413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:43.998415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:43.998581 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:43.999037 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:43.999043 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:43.999211 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 14:42:43.999315 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 14:42:43.999538 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":177,"threads_started":1}
I20260619 14:42:44.000010 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:43.986315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53840 (local address 127.0.98.62:41483)
0619 14:42:43.986645 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:43.986654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:43.987053 (+ 399us) server_negotiation.cc:407] Connection header received
0619 14:42:43.987150 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:43.987153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:43.987238 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:43.987346 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:43.988862 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.989466 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.990947 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:43.991262 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:43.991417 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:43.994845 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:43.994850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:43.994854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:43.994901 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:43.997553 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:43.998175 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:43.998179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:43.998181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:43.998282 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:43.998683 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:43.998686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:43.998688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:43.998929 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:43.999031 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:43.999394 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 14:42:43.999643 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
I20260619 14:42:44.004863 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53840:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:42:44.005522 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:42:44.010632 1784 tablet_service.cc:1467] Processing CreateTablet for tablet d207c498acad4429a23422d6812ea24c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ee397769c47d46aca959a510c8d40119]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:42:44.010579 1644 tablet_service.cc:1467] Processing CreateTablet for tablet d207c498acad4429a23422d6812ea24c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ee397769c47d46aca959a510c8d40119]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:42:44.010828 1924 tablet_service.cc:1467] Processing CreateTablet for tablet d207c498acad4429a23422d6812ea24c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ee397769c47d46aca959a510c8d40119]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:42:44.011044 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d207c498acad4429a23422d6812ea24c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:44.011050 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d207c498acad4429a23422d6812ea24c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:44.011137 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d207c498acad4429a23422d6812ea24c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:44.013927 2013 tablet_bootstrap.cc:492] T d207c498acad4429a23422d6812ea24c P 2e60821de1914d60bd786b394290d9b9: Bootstrap starting.
I20260619 14:42:44.013942 2016 tablet_bootstrap.cc:492] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21: Bootstrap starting.
I20260619 14:42:44.014849 2016 tablet_bootstrap.cc:654] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:44.014850 2013 tablet_bootstrap.cc:654] T d207c498acad4429a23422d6812ea24c P 2e60821de1914d60bd786b394290d9b9: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:44.016613 2013 tablet_bootstrap.cc:492] T d207c498acad4429a23422d6812ea24c P 2e60821de1914d60bd786b394290d9b9: No bootstrap required, opened a new log
I20260619 14:42:44.016645 2015 tablet_bootstrap.cc:492] T d207c498acad4429a23422d6812ea24c P 4d84caf110594070a00f6eb05caec487: Bootstrap starting.
I20260619 14:42:44.016731 2013 ts_tablet_manager.cc:1397] T d207c498acad4429a23422d6812ea24c P 2e60821de1914d60bd786b394290d9b9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:42:44.017172 2016 tablet_bootstrap.cc:492] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21: No bootstrap required, opened a new log
I20260619 14:42:44.017251 2016 ts_tablet_manager.cc:1397] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:42:44.017177 2013 raft_consensus.cc:348] T d207c498acad4429a23422d6812ea24c P 2e60821de1914d60bd786b394290d9b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } }
I20260619 14:42:44.017354 2013 raft_consensus.cc:374] T d207c498acad4429a23422d6812ea24c P 2e60821de1914d60bd786b394290d9b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:44.017395 2013 raft_consensus.cc:729] T d207c498acad4429a23422d6812ea24c P 2e60821de1914d60bd786b394290d9b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e60821de1914d60bd786b394290d9b9, State: Initialized, Role: FOLLOWER
I20260619 14:42:44.017434 2015 tablet_bootstrap.cc:654] T d207c498acad4429a23422d6812ea24c P 4d84caf110594070a00f6eb05caec487: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:44.017525 2013 consensus_queue.cc:260] T d207c498acad4429a23422d6812ea24c P 2e60821de1914d60bd786b394290d9b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } }
I20260619 14:42:44.017732 2016 raft_consensus.cc:348] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } }
I20260619 14:42:44.017820 2013 ts_tablet_manager.cc:1428] T d207c498acad4429a23422d6812ea24c P 2e60821de1914d60bd786b394290d9b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:42:44.017851 2016 raft_consensus.cc:374] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:44.017772 2033 tablet_replica.cc:439] T d207c498acad4429a23422d6812ea24c P 2e60821de1914d60bd786b394290d9b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } } }
I20260619 14:42:44.017882 2016 raft_consensus.cc:729] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21eb4bf8edc74177a4715450a2caef21, State: Initialized, Role: FOLLOWER
I20260619 14:42:44.017935 2033 tablet_replica.cc:442] T d207c498acad4429a23422d6812ea24c P 2e60821de1914d60bd786b394290d9b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:42:44.018005 2016 consensus_queue.cc:260] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } }
I20260619 14:42:44.018359 2021 tablet_replica.cc:439] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } } }
I20260619 14:42:44.018517 2021 tablet_replica.cc:442] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:42:44.019104 2015 tablet_bootstrap.cc:492] T d207c498acad4429a23422d6812ea24c P 4d84caf110594070a00f6eb05caec487: No bootstrap required, opened a new log
I20260619 14:42:44.019168 2016 ts_tablet_manager.cc:1428] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:42:44.019191 2015 ts_tablet_manager.cc:1397] T d207c498acad4429a23422d6812ea24c P 4d84caf110594070a00f6eb05caec487: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 14:42:44.019675 2015 raft_consensus.cc:348] T d207c498acad4429a23422d6812ea24c P 4d84caf110594070a00f6eb05caec487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } }
I20260619 14:42:44.019802 2015 raft_consensus.cc:374] T d207c498acad4429a23422d6812ea24c P 4d84caf110594070a00f6eb05caec487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:44.019842 2015 raft_consensus.cc:729] T d207c498acad4429a23422d6812ea24c P 4d84caf110594070a00f6eb05caec487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d84caf110594070a00f6eb05caec487, State: Initialized, Role: FOLLOWER
I20260619 14:42:44.019963 2015 consensus_queue.cc:260] T d207c498acad4429a23422d6812ea24c P 4d84caf110594070a00f6eb05caec487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } }
I20260619 14:42:44.020258 2015 ts_tablet_manager.cc:1428] T d207c498acad4429a23422d6812ea24c P 4d84caf110594070a00f6eb05caec487: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:42:44.020272 2020 tablet_replica.cc:439] T d207c498acad4429a23422d6812ea24c P 4d84caf110594070a00f6eb05caec487: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } } }
I20260619 14:42:44.020450 2020 tablet_replica.cc:442] T d207c498acad4429a23422d6812ea24c P 4d84caf110594070a00f6eb05caec487: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:42:44.262488 2021 raft_consensus.cc:482] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:42:44.262665 2021 raft_consensus.cc:504] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } }
I20260619 14:42:44.264007 2021 leader_election.cc:288] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d84caf110594070a00f6eb05caec487 (127.0.98.1:43571), 2e60821de1914d60bd786b394290d9b9 (127.0.98.3:43901)
I20260619 14:42:44.268370 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:44.264447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38369 (local address 127.0.98.3:43901)
0619 14:42:44.264662 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:44.264667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:44.264822 (+ 155us) server_negotiation.cc:407] Connection header received
0619 14:42:44.264982 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:44.264985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:44.265072 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:44.265167 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:44.265386 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.266074 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.267135 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.267807 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.267963 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:44.268044 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:44.268167 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 14:42:44.268234 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260619 14:42:44.268392 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:44.264338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43901 (local address 127.0.98.2:38369)
0619 14:42:44.264719 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:44.264735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:44.264841 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:44.265190 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:44.265192 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:44.265206 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:44.265276 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:44.265280 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.266204 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.266207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:44.267023 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:44.267031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.267942 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.267946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:44.268036 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:44.268067 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 14:42:44.268153 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 14:42:44.268225 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":19,"thread_start_us":118,"threads_started":1}
I20260619 14:42:44.268632 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:44.264609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60541 (local address 127.0.98.1:43571)
0619 14:42:44.264828 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:44.264832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:44.264847 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:42:44.264905 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:44.264909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:44.264956 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:44.265036 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:44.265417 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.265911 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.267310 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.267996 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.268099 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:44.268160 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:44.268436 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 14:42:44.268512 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":59,"threads_started":1}
I20260619 14:42:44.268616 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:44.264211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43571 (local address 127.0.98.2:60541)
0619 14:42:44.264537 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:44.264552 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:44.264663 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:44.265086 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:44.265090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:44.265113 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:44.265244 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:44.265251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.266019 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.266022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:44.267188 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:44.267198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.268209 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.268212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:44.268316 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:44.268342 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:42:44.268407 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:42:44.268472 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":80,"thread_start_us":133,"threads_started":1}
I20260619 14:42:44.269413 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d207c498acad4429a23422d6812ea24c" candidate_uuid: "21eb4bf8edc74177a4715450a2caef21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e60821de1914d60bd786b394290d9b9" is_pre_election: true
I20260619 14:42:44.269387 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d207c498acad4429a23422d6812ea24c" candidate_uuid: "21eb4bf8edc74177a4715450a2caef21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d84caf110594070a00f6eb05caec487" is_pre_election: true
I20260619 14:42:44.269551 1944 raft_consensus.cc:2454] T d207c498acad4429a23422d6812ea24c P 2e60821de1914d60bd786b394290d9b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21eb4bf8edc74177a4715450a2caef21 in term 0.
I20260619 14:42:44.269551 1663 raft_consensus.cc:2454] T d207c498acad4429a23422d6812ea24c P 4d84caf110594070a00f6eb05caec487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21eb4bf8edc74177a4715450a2caef21 in term 0.
I20260619 14:42:44.269933 1738 leader_election.cc:302] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21eb4bf8edc74177a4715450a2caef21, 2e60821de1914d60bd786b394290d9b9, 4d84caf110594070a00f6eb05caec487; no voters:
I20260619 14:42:44.270215 2021 raft_consensus.cc:2781] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:42:44.270290 2021 raft_consensus.cc:482] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:42:44.270344 2021 raft_consensus.cc:3037] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:44.271651 2021 raft_consensus.cc:504] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } }
I20260619 14:42:44.271976 2021 leader_election.cc:288] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 [CANDIDATE]: Term 1 election: Requested vote from peers 4d84caf110594070a00f6eb05caec487 (127.0.98.1:43571), 2e60821de1914d60bd786b394290d9b9 (127.0.98.3:43901)
I20260619 14:42:44.272360 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d207c498acad4429a23422d6812ea24c" candidate_uuid: "21eb4bf8edc74177a4715450a2caef21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d84caf110594070a00f6eb05caec487"
I20260619 14:42:44.272360 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d207c498acad4429a23422d6812ea24c" candidate_uuid: "21eb4bf8edc74177a4715450a2caef21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e60821de1914d60bd786b394290d9b9"
I20260619 14:42:44.272490 1663 raft_consensus.cc:3037] T d207c498acad4429a23422d6812ea24c P 4d84caf110594070a00f6eb05caec487 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:44.272490 1944 raft_consensus.cc:3037] T d207c498acad4429a23422d6812ea24c P 2e60821de1914d60bd786b394290d9b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:44.273589 1663 raft_consensus.cc:2454] T d207c498acad4429a23422d6812ea24c P 4d84caf110594070a00f6eb05caec487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21eb4bf8edc74177a4715450a2caef21 in term 1.
I20260619 14:42:44.273593 1944 raft_consensus.cc:2454] T d207c498acad4429a23422d6812ea24c P 2e60821de1914d60bd786b394290d9b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21eb4bf8edc74177a4715450a2caef21 in term 1.
I20260619 14:42:44.273897 1736 leader_election.cc:302] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eb4bf8edc74177a4715450a2caef21, 2e60821de1914d60bd786b394290d9b9; no voters:
I20260619 14:42:44.274080 2021 raft_consensus.cc:2781] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:42:44.274303 2021 raft_consensus.cc:686] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 [term 1 LEADER]: Becoming Leader. State: Replica: 21eb4bf8edc74177a4715450a2caef21, State: Running, Role: LEADER
I20260619 14:42:44.274560 2021 consensus_queue.cc:237] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } }
I20260619 14:42:44.275671 2059 tablet_replica.cc:439] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21: TxnStatusTablet state changed. Reason: New leader 21eb4bf8edc74177a4715450a2caef21. Latest consensus state: current_term: 1 leader_uuid: "21eb4bf8edc74177a4715450a2caef21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } } }
I20260619 14:42:44.275775 2059 tablet_replica.cc:442] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:42:44.276054 2061 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:42:44.276223 2061 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:42:44.277014 1507 catalog_manager.cc:5495] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21eb4bf8edc74177a4715450a2caef21 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "21eb4bf8edc74177a4715450a2caef21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } health_report { overall_health: UNKNOWN } } }
I20260619 14:42:44.319929 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:44.315379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.0.98.2:41561)
0619 14:42:44.315539 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:44.315544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:44.315561 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:42:44.315724 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:44.315728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:44.315799 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:44.315914 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:42:44.316246 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.317141 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.318120 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.318495 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.318601 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:44.318872 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:42:44.319422 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:42:44.319574 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:44.319685 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 14:42:44.319744 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 14:42:44.319960 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:44.315283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41561 (local address 127.0.0.1:43416)
0619 14:42:44.315416 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:44.315457 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:44.315564 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:44.316017 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:44.316024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:44.316038 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:42:44.316149 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:44.316154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.317267 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.317270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:44.317998 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:44.318007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.318615 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.318618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:44.318683 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:44.318726 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:42:44.319558 (+ 832us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:42:44.319574 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 14:42:44.319649 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 14:42:44.319780 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 14:42:44.323140 1663 raft_consensus.cc:1261] T d207c498acad4429a23422d6812ea24c P 4d84caf110594070a00f6eb05caec487 [term 1 FOLLOWER]: Refusing update from remote peer 21eb4bf8edc74177a4715450a2caef21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:42:44.323392 1944 raft_consensus.cc:1261] T d207c498acad4429a23422d6812ea24c P 2e60821de1914d60bd786b394290d9b9 [term 1 FOLLOWER]: Refusing update from remote peer 21eb4bf8edc74177a4715450a2caef21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:42:44.323688 2059 consensus_queue.cc:1035] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:42:44.324103 2021 consensus_queue.cc:1035] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:42:44.325299 2020 tablet_replica.cc:439] T d207c498acad4429a23422d6812ea24c P 4d84caf110594070a00f6eb05caec487: TxnStatusTablet state changed. Reason: New leader 21eb4bf8edc74177a4715450a2caef21. Latest consensus state: current_term: 1 leader_uuid: "21eb4bf8edc74177a4715450a2caef21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } } }
I20260619 14:42:44.325431 2020 tablet_replica.cc:442] T d207c498acad4429a23422d6812ea24c P 4d84caf110594070a00f6eb05caec487: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:42:44.325774 2033 tablet_replica.cc:439] T d207c498acad4429a23422d6812ea24c P 2e60821de1914d60bd786b394290d9b9: TxnStatusTablet state changed. Reason: New leader 21eb4bf8edc74177a4715450a2caef21. Latest consensus state: current_term: 1 leader_uuid: "21eb4bf8edc74177a4715450a2caef21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } } }
I20260619 14:42:44.325892 2033 tablet_replica.cc:442] T d207c498acad4429a23422d6812ea24c P 2e60821de1914d60bd786b394290d9b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:42:44.328084 2021 tablet_replica.cc:439] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21eb4bf8edc74177a4715450a2caef21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } } }
I20260619 14:42:44.328200 2021 tablet_replica.cc:442] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:42:44.329087 2059 tablet_replica.cc:439] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21eb4bf8edc74177a4715450a2caef21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } } }
I20260619 14:42:44.329190 2059 tablet_replica.cc:442] T d207c498acad4429a23422d6812ea24c P 21eb4bf8edc74177a4715450a2caef21: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:42:44.330159 2020 tablet_replica.cc:439] T d207c498acad4429a23422d6812ea24c P 4d84caf110594070a00f6eb05caec487: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21eb4bf8edc74177a4715450a2caef21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } } }
I20260619 14:42:44.330278 2020 tablet_replica.cc:442] T d207c498acad4429a23422d6812ea24c P 4d84caf110594070a00f6eb05caec487: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:42:44.331782 2033 tablet_replica.cc:439] T d207c498acad4429a23422d6812ea24c P 2e60821de1914d60bd786b394290d9b9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21eb4bf8edc74177a4715450a2caef21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d84caf110594070a00f6eb05caec487" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43571 } } peers { permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 } } peers { permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 } } }
I20260619 14:42:44.331904 2033 tablet_replica.cc:442] T d207c498acad4429a23422d6812ea24c P 2e60821de1914d60bd786b394290d9b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:42:44.338496 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:44.334381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.0.98.1:43571)
0619 14:42:44.334635 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:44.334640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:44.334660 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:42:44.334879 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:44.334882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:44.334948 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:44.335064 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:42:44.335309 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.335948 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.336896 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.337163 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.337282 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:44.337574 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:42:44.338033 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:42:44.338179 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:44.338225 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 14:42:44.338287 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260619 14:42:44.339810 2020 consensus_queue.cc:1035] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eb4bf8edc74177a4715450a2caef21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:42:44.344512 2044 consensus_queue.cc:1035] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e60821de1914d60bd786b394290d9b9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:42:44.351912 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:44.348597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54707 (local address 127.0.98.2:41561)
0619 14:42:44.348744 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:44.348748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:44.348761 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:42:44.348809 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:44.348811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:44.348848 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:44.348937 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:42:44.349203 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.349758 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.350739 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.350996 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.351114 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:44.351417 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:42:44.351523 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:42:44.351612 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:44.351722 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 14:42:44.351772 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:42:44.351941 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:44.348198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41561 (local address 127.0.98.1:54707)
0619 14:42:44.348503 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:44.348521 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:44.348616 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:44.348948 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:44.348951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:44.348959 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:42:44.349081 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:44.349093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.349898 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.349902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:44.350602 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:44.350610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.351115 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.351119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:44.351223 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:44.351277 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:42:44.351629 (+ 352us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:42:44.351643 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 14:42:44.351711 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:42:44.351794 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":118,"threads_started":1}
I20260619 14:42:44.380225 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:44.374837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58021 (local address 127.0.98.1:43571)
0619 14:42:44.374981 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:44.374986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:44.375807 (+ 821us) server_negotiation.cc:407] Connection header received
0619 14:42:44.376012 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:44.376015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:44.376071 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:44.376153 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:42:44.376831 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.377713 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.378768 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.379134 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.379266 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:44.379656 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:42:44.379778 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:42:44.379863 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:44.380003 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 14:42:44.380061 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:42:44.380246 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:44.374791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43571 (local address 127.0.98.2:58021)
0619 14:42:44.375716 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:44.375733 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:44.375887 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:44.376166 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:44.376169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:44.376189 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:42:44.376702 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:44.376711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.377859 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.377863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:44.378569 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:44.378579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.379277 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.379281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:44.379399 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:44.379496 (+ 97us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:42:44.379883 (+ 387us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:42:44.379891 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:42:44.379971 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 14:42:44.380058 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":127,"threads_started":1}
I20260619 14:42:44.405225 1505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:42:44.405426 1505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:42:44.408016 1505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9fa752c94804242a7667304400c19b5: Sending DeleteTablet for 3 replicas of tablet 3e9a10e7ce1d4605a2f79a8ac774d7c6
I20260619 14:42:44.408608 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e9a10e7ce1d4605a2f79a8ac774d7c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:42:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45470
I20260619 14:42:44.408790 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e9a10e7ce1d4605a2f79a8ac774d7c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:42:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43414
I20260619 14:42:44.409134 2087 tablet_replica.cc:330] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487: stopping tablet replica
I20260619 14:42:44.409318 2087 raft_consensus.cc:2229] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:42:44.409608 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e9a10e7ce1d4605a2f79a8ac774d7c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:42:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37708
I20260619 14:42:44.409683 2088 tablet_replica.cc:330] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 21eb4bf8edc74177a4715450a2caef21: stopping tablet replica
I20260619 14:42:44.409804 2088 raft_consensus.cc:2229] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 21eb4bf8edc74177a4715450a2caef21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:42:44.409919 2088 raft_consensus.cc:2258] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 21eb4bf8edc74177a4715450a2caef21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:42:44.410655 2087 raft_consensus.cc:2258] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:42:44.411209 2088 ts_tablet_manager.cc:1905] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 21eb4bf8edc74177a4715450a2caef21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:42:44.414466 2088 ts_tablet_manager.cc:1918] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 21eb4bf8edc74177a4715450a2caef21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:42:44.414558 2088 log.cc:1196] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 21eb4bf8edc74177a4715450a2caef21: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/3e9a10e7ce1d4605a2f79a8ac774d7c6
I20260619 14:42:44.414841 2088 ts_tablet_manager.cc:1939] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 21eb4bf8edc74177a4715450a2caef21: Deleting consensus metadata
I20260619 14:42:44.415683 2089 tablet_replica.cc:330] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 2e60821de1914d60bd786b394290d9b9: stopping tablet replica
I20260619 14:42:44.415669 1493 catalog_manager.cc:4841] TS 21eb4bf8edc74177a4715450a2caef21 (127.0.98.2:41561): tablet 3e9a10e7ce1d4605a2f79a8ac774d7c6 (table test-table [id=af6591052ce74d3baa5eb9e5985882a1]) successfully deleted
I20260619 14:42:44.415843 2089 raft_consensus.cc:2229] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 2e60821de1914d60bd786b394290d9b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:42:44.415979 2089 raft_consensus.cc:2258] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 2e60821de1914d60bd786b394290d9b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:42:44.416188 2087 ts_tablet_manager.cc:1905] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:42:44.417793 2089 ts_tablet_manager.cc:1905] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 2e60821de1914d60bd786b394290d9b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:42:44.419194 2087 ts_tablet_manager.cc:1918] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:42:44.419287 2087 log.cc:1196] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-0/wal/wals/3e9a10e7ce1d4605a2f79a8ac774d7c6
I20260619 14:42:44.419636 2087 ts_tablet_manager.cc:1939] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 4d84caf110594070a00f6eb05caec487: Deleting consensus metadata
I20260619 14:42:44.420403 1494 catalog_manager.cc:4841] TS 4d84caf110594070a00f6eb05caec487 (127.0.98.1:43571): tablet 3e9a10e7ce1d4605a2f79a8ac774d7c6 (table test-table [id=af6591052ce74d3baa5eb9e5985882a1]) successfully deleted
I20260619 14:42:44.421804 2089 ts_tablet_manager.cc:1918] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 2e60821de1914d60bd786b394290d9b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:42:44.421897 2089 log.cc:1196] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 2e60821de1914d60bd786b394290d9b9: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/3e9a10e7ce1d4605a2f79a8ac774d7c6
I20260619 14:42:44.422194 2089 ts_tablet_manager.cc:1939] T 3e9a10e7ce1d4605a2f79a8ac774d7c6 P 2e60821de1914d60bd786b394290d9b9: Deleting consensus metadata
I20260619 14:42:44.423000 1492 catalog_manager.cc:4841] TS 2e60821de1914d60bd786b394290d9b9 (127.0.98.3:43901): tablet 3e9a10e7ce1d4605a2f79a8ac774d7c6 (table test-table [id=af6591052ce74d3baa5eb9e5985882a1]) successfully deleted
Jun 19 14:42:44 dist-test-slave-cr22 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880163, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:42:44.435422 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:44.419879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45494 (local address 127.0.98.1:43571)
0619 14:42:44.420021 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:44.420026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:44.423074 (+ 3048us) server_negotiation.cc:407] Connection header received
0619 14:42:44.423293 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:44.423297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:44.423352 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:44.423468 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:44.424595 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.425466 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.426642 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.427037 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.427148 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:44.431036 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:44.431046 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:44.431060 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:44.431090 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:44.433215 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:44.433738 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:44.433745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:44.433747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:44.433850 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:44.434181 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:44.434184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:44.434186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:44.434504 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:44.434688 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:44.435092 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 14:42:44.435225 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 14:42:44.436378 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45494
I20260619 14:42:44.448663 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:44.438796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53852 (local address 127.0.98.62:41483)
0619 14:42:44.438950 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:44.438956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:44.439284 (+ 328us) server_negotiation.cc:407] Connection header received
0619 14:42:44.439560 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:44.439564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:44.439620 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:44.439681 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:44.440584 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.441406 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.442538 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:44.442886 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:44.442968 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:44.444739 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:44.444742 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:44.444745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:44.444771 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:44.446632 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:44.447131 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:44.447135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:44.447137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:44.447225 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:44.447616 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:44.447619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:44.447621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:44.447846 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:44.447965 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:44.448353 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 14:42:44.448490 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 14:42:44.449467 1505 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53852
I20260619 14:42:44.451417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 1579
I20260619 14:42:44.460561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 1719
I20260619 14:42:44.467648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 1859
I20260619 14:42:44.474030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 1475
2026-06-19T14:42:44Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3640 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:42:44 dist-test-slave-cr22 krb5kdc[2096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:42:44 dist-test-slave-cr22 krb5kdc[2096](info): set up 2 sockets
Jun 19 14:42:44 dist-test-slave-cr22 krb5kdc[2096](info): commencing operation
krb5kdc: starting...
W20260619 14:42:45.607389 392 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:42:45 dist-test-slave-cr22 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:42:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:42:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:42:45.735168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35575
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:45.847554 2109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:45.847898 2109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:45.847981 2109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:45.852191 2109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:42:45.852285 2109 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:42:45.852317 2109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:45.852361 2109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:45.852397 2109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:42:45.852432 2109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:42:45.857878 2109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35575
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:45.859017 2109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:45.860235 2109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:42:45.868887 2109 server_base.cc:1034] running on GCE node
W20260619 14:42:45.868839 2117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:45.868896 2115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:45.868839 2114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:45.869727 2109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:45.870934 2109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:45.872119 2109 hybrid_clock.cc:648] HybridClock initialized: now 1781880165872091 us; error 41 us; skew 500 ppm
Jun 19 14:42:45 dist-test-slave-cr22 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:45.875059 2109 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:42:45.876063 2109 webserver.cc:460] Webserver started at http://127.0.98.62:34393/ using document root <none> and password file <none>
I20260619 14:42:45.876601 2109 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:45.876675 2109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:45.876843 2109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:45.878827 2109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "ee86775e38c646c5a6796977f2f537bb"
format_stamp: "Formatted at 2026-06-19 14:42:45 on dist-test-slave-cr22"
server_key: "25936494ceb637249ba5e56a49761447"
server_key_iv: "1e8b43d520f33a98f7a66ea4173bca72"
server_key_version: "encryptionkey@0"
I20260619 14:42:45.879351 2109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee86775e38c646c5a6796977f2f537bb"
format_stamp: "Formatted at 2026-06-19 14:42:45 on dist-test-slave-cr22"
server_key: "25936494ceb637249ba5e56a49761447"
server_key_iv: "1e8b43d520f33a98f7a66ea4173bca72"
server_key_version: "encryptionkey@0"
I20260619 14:42:45.883225 2109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 14:42:45.885682 2124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:45.887034 2109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:42:45.887172 2109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "ee86775e38c646c5a6796977f2f537bb"
format_stamp: "Formatted at 2026-06-19 14:42:45 on dist-test-slave-cr22"
server_key: "25936494ceb637249ba5e56a49761447"
server_key_iv: "1e8b43d520f33a98f7a66ea4173bca72"
server_key_version: "encryptionkey@0"
I20260619 14:42:45.887286 2109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:45.909895 2109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:45.913307 2109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:45.913518 2109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:45.921720 2109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35575
I20260619 14:42:45.921738 2186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35575 every 8 connection(s)
I20260619 14:42:45.922734 2109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:42:45.925644 2187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:45.931172 2187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb: Bootstrap starting.
I20260619 14:42:45.932025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 2109
I20260619 14:42:45.932169 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:42:45.932428 392 external_mini_cluster.cc:1428] Setting key 0fb94ebee49c1d0eb18fcf40635c3e6d
I20260619 14:42:45.934000 2187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:45.934712 2187 log.cc:824] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb: Log is configured to *not* fsync() on all Append() calls
I20260619 14:42:45.936944 2187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb: No bootstrap required, opened a new log
I20260619 14:42:45.941421 2187 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee86775e38c646c5a6796977f2f537bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35575 } }
I20260619 14:42:45.941679 2187 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Jun 19 14:42:45 dist-test-slave-cr22 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:45.941763 2187 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee86775e38c646c5a6796977f2f537bb, State: Initialized, Role: FOLLOWER
I20260619 14:42:45.942286 2187 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee86775e38c646c5a6796977f2f537bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35575 } }
I20260619 14:42:45.942474 2187 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:42:45.942540 2187 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:42:45.942636 2187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:45.943578 2187 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee86775e38c646c5a6796977f2f537bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35575 } }
I20260619 14:42:45.943979 2187 leader_election.cc:302] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee86775e38c646c5a6796977f2f537bb; no voters:
I20260619 14:42:45.944303 2187 leader_election.cc:288] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:42:45.944430 2192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:42:45.944674 2192 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb [term 1 LEADER]: Becoming Leader. State: Replica: ee86775e38c646c5a6796977f2f537bb, State: Running, Role: LEADER
I20260619 14:42:45.945138 2192 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee86775e38c646c5a6796977f2f537bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35575 } }
I20260619 14:42:45.945663 2187 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:42:45.946993 2194 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee86775e38c646c5a6796977f2f537bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee86775e38c646c5a6796977f2f537bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35575 } } }
I20260619 14:42:45.947149 2194 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb [sys.catalog]: This master's current role is: LEADER
I20260619 14:42:45.947650 2193 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee86775e38c646c5a6796977f2f537bb. Latest consensus state: current_term: 1 leader_uuid: "ee86775e38c646c5a6796977f2f537bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee86775e38c646c5a6796977f2f537bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35575 } } }
I20260619 14:42:45.947860 2193 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb [sys.catalog]: This master's current role is: LEADER
I20260619 14:42:45.949592 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:45.933575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33758 (local address 127.0.98.62:35575)
0619 14:42:45.934032 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:45.934047 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:45.934083 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:42:45.934787 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:45.934804 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:45.935236 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:45.935702 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:45.936454 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:45.937565 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:45.938841 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:45.939345 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:45.939560 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:45.942362 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:45.942367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:45.942387 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:45.942416 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:45.945003 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:45.945680 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:45.945686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:45.945693 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:45.945845 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:45.946180 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:45.946184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:45.946186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:45.946789 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:45.947018 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:45.948294 (+ 1276us) server_negotiation.cc:299] Negotiation successful
0619 14:42:45.948653 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":157,"threads_started":1}
W20260619 14:42:45.951479 2207 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:42:45.951828 2207 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:42:45.951910 2203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:42:45.952622 2203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:42:45.957816 2203 catalog_manager.cc:1347] Generated new cluster ID: ef9275cfdfb3409fbcf46a192b305588
I20260619 14:42:45.957877 2203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:42:45.971840 2203 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:42:45.973189 2203 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:42:45.987052 2203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb: Generated new TSK 0
I20260619 14:42:45.988004 2203 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:42:46.021349 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35575
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:46.134295 2213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:46.134558 2213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:46.134608 2213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:46.138684 2213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:46.138770 2213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:46.138813 2213 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:42:46.138828 2213 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:42:46.138861 2213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:42:46.144145 2213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:46.145119 2213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:46.146134 2213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:46.156002 2219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:46.156002 2221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:46.156124 2218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:46.156252 2213 server_base.cc:1034] running on GCE node
I20260619 14:42:46.156744 2213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:46.157380 2213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:46.158584 2213 hybrid_clock.cc:648] HybridClock initialized: now 1781880166158548 us; error 48 us; skew 500 ppm
Jun 19 14:42:46 dist-test-slave-cr22 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:46.161441 2213 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:42:46.162360 2213 webserver.cc:460] Webserver started at http://127.0.98.1:33759/ using document root <none> and password file <none>
I20260619 14:42:46.162928 2213 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:46.163018 2213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:46.163255 2213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:46.165374 2213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "28125b7821d2490897d0b36297a2b239"
format_stamp: "Formatted at 2026-06-19 14:42:46 on dist-test-slave-cr22"
server_key: "a5fdb9a50095a717e11971b434bfc66d"
server_key_iv: "8ee02359834464204a3ced0df7d66b84"
server_key_version: "encryptionkey@0"
I20260619 14:42:46.165925 2213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "28125b7821d2490897d0b36297a2b239"
format_stamp: "Formatted at 2026-06-19 14:42:46 on dist-test-slave-cr22"
server_key: "a5fdb9a50095a717e11971b434bfc66d"
server_key_iv: "8ee02359834464204a3ced0df7d66b84"
server_key_version: "encryptionkey@0"
I20260619 14:42:46.169675 2213 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:42:46.172165 2228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:46.173552 2213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:42:46.173873 2213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "28125b7821d2490897d0b36297a2b239"
format_stamp: "Formatted at 2026-06-19 14:42:46 on dist-test-slave-cr22"
server_key: "a5fdb9a50095a717e11971b434bfc66d"
server_key_iv: "8ee02359834464204a3ced0df7d66b84"
server_key_version: "encryptionkey@0"
I20260619 14:42:46.174006 2213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:46.209568 2213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:46.213060 2213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:46.213310 2213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:46.214772 2213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:46.214852 2213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:46.214927 2213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:46.214972 2213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:42:46 dist-test-slave-cr22 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:46.225896 2213 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45369
I20260619 14:42:46.226250 2343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45369 every 8 connection(s)
I20260619 14:42:46.226830 2213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:42:46.228837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 2213
I20260619 14:42:46.228971 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:42:46.229197 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.215936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57885 (local address 127.0.98.62:35575)
0619 14:42:46.216087 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:46.216092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:46.217017 (+ 925us) server_negotiation.cc:407] Connection header received
0619 14:42:46.218369 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:46.218374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:46.218460 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:46.218560 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:46.220236 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.220838 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.222014 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.222317 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.222448 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.225122 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:46.225125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:46.225128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:46.225163 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:46.226764 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:46.227279 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:46.227284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:46.227286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:46.227374 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:46.227824 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:46.227827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:46.227829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:46.228026 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:46.228105 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:46.228834 (+ 729us) server_negotiation.cc:299] Negotiation successful
0619 14:42:46.229015 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:42:46.229290 392 external_mini_cluster.cc:1428] Setting key 8fd7938f2abf8d3dcb335b9e1e95ec47
I20260619 14:42:46.229851 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.216338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35575 (local address 127.0.98.1:57885)
0619 14:42:46.216837 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:46.216894 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:46.218075 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:46.218755 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:46.218763 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:46.219271 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:46.220063 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.220083 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.220973 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.220977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.221875 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.221884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.222499 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.222503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.222660 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.223294 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:46.223312 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:46.224958 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:46.226881 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:46.226886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:46.226900 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:46.227167 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:46.227527 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:46.227532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:46.227534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:46.227715 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:46.228152 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:46.228157 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:46.228389 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 14:42:46.228554 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 14:42:46.228797 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":117,"threads_started":1}
I20260619 14:42:46.231314 2344 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35575
I20260619 14:42:46.231518 2344 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:46.231992 2344 heartbeater.cc:511] Master 127.0.98.62:35575 requested a full tablet report, sending...
I20260619 14:42:46.233700 2137 ts_manager.cc:194] Registered new tserver with Master: 28125b7821d2490897d0b36297a2b239 (127.0.98.1:45369)
I20260619 14:42:46.235031 2137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57885
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:42:46.241881 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.233861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45887 (local address 127.0.98.62:35575)
0619 14:42:46.233978 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:46.233982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:46.234275 (+ 293us) server_negotiation.cc:407] Connection header received
0619 14:42:46.234390 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:46.234393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:46.234435 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:46.234520 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:46.235099 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.235861 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.236976 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.237177 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.237255 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.238561 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:46.238565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:46.238567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:46.238587 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:46.240109 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:46.240618 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:46.240621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:46.240623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:46.240715 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:46.241034 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:46.241036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:46.241038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:46.241216 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:46.241305 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:46.241651 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 14:42:46.241749 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:42:46.241930 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.233791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35575 (local address 127.0.98.1:45887)
0619 14:42:46.234174 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:46.234191 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:46.234297 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:46.234556 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:46.234562 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:46.234816 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:46.234968 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.234979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.235996 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.235999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.236861 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.236870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.237288 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.237291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.237383 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.237892 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:46.237911 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:46.238420 (+ 509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:46.240222 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:46.240227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:46.240232 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:46.240485 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:46.240824 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:46.240829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:46.240831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:46.240931 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:46.241319 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:46.241322 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:46.241425 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 14:42:46.241554 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 14:42:46.241711 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:42:46.262499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35575
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:46.372195 2353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:46.372448 2353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:46.372493 2353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:46.376547 2353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:46.376608 2353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:46.376694 2353 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:42:46.376724 2353 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:42:46.376782 2353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:42:46.382129 2353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:46.383203 2353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:46.384282 2353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:46.394702 2359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:46.394810 2353 server_base.cc:1034] running on GCE node
W20260619 14:42:46.394711 2358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:46.394835 2361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:46.395387 2353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:46.396116 2353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:46.397351 2353 hybrid_clock.cc:648] HybridClock initialized: now 1781880166397300 us; error 60 us; skew 500 ppm
Jun 19 14:42:46 dist-test-slave-cr22 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:46.400542 2353 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:42:46.401537 2353 webserver.cc:460] Webserver started at http://127.0.98.2:45477/ using document root <none> and password file <none>
I20260619 14:42:46.402117 2353 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:46.402209 2353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:46.402446 2353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:46.404580 2353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "00177649bdaf493a8a03c6ebba1d2d09"
format_stamp: "Formatted at 2026-06-19 14:42:46 on dist-test-slave-cr22"
server_key: "ea8a9ad5d14b8646e4ba10690151461f"
server_key_iv: "cace06dd886ef0e69b153bacfb784277"
server_key_version: "encryptionkey@0"
I20260619 14:42:46.405145 2353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "00177649bdaf493a8a03c6ebba1d2d09"
format_stamp: "Formatted at 2026-06-19 14:42:46 on dist-test-slave-cr22"
server_key: "ea8a9ad5d14b8646e4ba10690151461f"
server_key_iv: "cace06dd886ef0e69b153bacfb784277"
server_key_version: "encryptionkey@0"
I20260619 14:42:46.408877 2353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:42:46.411221 2368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:46.412499 2353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:42:46.412645 2353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "00177649bdaf493a8a03c6ebba1d2d09"
format_stamp: "Formatted at 2026-06-19 14:42:46 on dist-test-slave-cr22"
server_key: "ea8a9ad5d14b8646e4ba10690151461f"
server_key_iv: "cace06dd886ef0e69b153bacfb784277"
server_key_version: "encryptionkey@0"
I20260619 14:42:46.412758 2353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:46.428341 2353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:46.431417 2353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:46.431627 2353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:46.433074 2353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:46.433145 2353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:46.433214 2353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:46.433264 2353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:42:46 dist-test-slave-cr22 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:46.444113 2353 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45007
I20260619 14:42:46.444214 2483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45007 every 8 connection(s)
I20260619 14:42:46.444988 2353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:42:46.447021 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.433863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58777 (local address 127.0.98.62:35575)
0619 14:42:46.434009 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:46.434014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:46.434934 (+ 920us) server_negotiation.cc:407] Connection header received
0619 14:42:46.436289 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:46.436294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:46.436355 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:46.436457 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:46.437540 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.438061 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.439410 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.439705 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.439819 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.442941 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:46.442944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:46.442947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:46.442993 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:46.444670 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:46.445278 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:46.445282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:46.445284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:46.445372 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:46.445826 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:46.445829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:46.445831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:46.446077 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:46.446167 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:46.446653 (+ 486us) server_negotiation.cc:299] Negotiation successful
0619 14:42:46.446871 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:42:46.447883 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.434278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35575 (local address 127.0.98.2:58777)
0619 14:42:46.434736 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:46.434793 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:46.435974 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:46.436639 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:46.436646 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:46.436964 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:46.437401 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.437413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.438181 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.438184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.439278 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.439289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.439854 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.439858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.440030 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.440672 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:46.440694 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:46.442764 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:46.444795 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:46.444801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:46.444815 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:46.445175 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:46.445496 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:46.445499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:46.445501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:46.445677 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:46.446185 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:46.446190 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:46.446405 (+ 215us) client_negotiation.cc:769] Sending connection context
0619 14:42:46.446561 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 14:42:46.446996 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":108,"threads_started":1}
I20260619 14:42:46.449045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 2353
I20260619 14:42:46.449141 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:42:46.449158 2484 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35575
I20260619 14:42:46.449295 2484 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:46.449437 392 external_mini_cluster.cc:1428] Setting key c0a0b0fffb61ac6cce903a432b7b6c35
I20260619 14:42:46.449848 2484 heartbeater.cc:511] Master 127.0.98.62:35575 requested a full tablet report, sending...
I20260619 14:42:46.450814 2137 ts_manager.cc:194] Registered new tserver with Master: 00177649bdaf493a8a03c6ebba1d2d09 (127.0.98.2:45007)
I20260619 14:42:46.451637 2137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58777
I20260619 14:42:46.459227 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.451532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51305 (local address 127.0.98.62:35575)
0619 14:42:46.451644 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:46.451651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:46.452015 (+ 364us) server_negotiation.cc:407] Connection header received
0619 14:42:46.452199 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:46.452202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:46.452247 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:46.452331 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:46.452926 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.453435 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.454464 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.454665 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.454740 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.456047 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:46.456050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:46.456052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:46.456071 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:46.457497 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:46.458022 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:46.458024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:46.458026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:46.458110 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:46.458437 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:46.458439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:46.458441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:46.458632 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:46.458712 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:46.459034 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 14:42:46.459118 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:42:46.459481 2490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.451459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35575 (local address 127.0.98.2:51305)
0619 14:42:46.451821 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:46.451961 (+ 140us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:46.452081 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:46.452364 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:46.452369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:46.452607 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:46.452777 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.452788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.453555 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.453558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.454335 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.454348 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.454782 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.454789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.454910 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.455397 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:46.455413 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:46.455918 (+ 505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:46.457607 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:46.457610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:46.457612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:46.457894 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:46.458211 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:46.458214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:46.458217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:46.458318 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:46.458748 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:46.458752 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:46.458862 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 14:42:46.458992 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 14:42:46.459172 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:42:46.480145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35575
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:46.593137 2493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:46.593380 2493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:46.593461 2493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:46.597584 2493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:46.597641 2493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:46.597687 2493 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:42:46.597726 2493 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:42:46.597792 2493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:42:46.603153 2493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:46.604206 2493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:46.605201 2493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:46.614701 2501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:46.614780 2493 server_base.cc:1034] running on GCE node
W20260619 14:42:46.614728 2498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:46.614710 2499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:46.615330 2493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:46.616044 2493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:46.617197 2493 hybrid_clock.cc:648] HybridClock initialized: now 1781880166617172 us; error 54 us; skew 500 ppm
Jun 19 14:42:46 dist-test-slave-cr22 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:46.620126 2493 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:42:46.620954 2493 webserver.cc:460] Webserver started at http://127.0.98.3:33975/ using document root <none> and password file <none>
I20260619 14:42:46.621495 2493 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:46.621579 2493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:46.621798 2493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:46.623843 2493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "6bdc477b990644489fce0187cd7b70ea"
format_stamp: "Formatted at 2026-06-19 14:42:46 on dist-test-slave-cr22"
server_key: "8ef67b283bfd184174a0dbeecbc50ec0"
server_key_iv: "f6ab040fc9209a430a5e500240c96858"
server_key_version: "encryptionkey@0"
I20260619 14:42:46.624393 2493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6bdc477b990644489fce0187cd7b70ea"
format_stamp: "Formatted at 2026-06-19 14:42:46 on dist-test-slave-cr22"
server_key: "8ef67b283bfd184174a0dbeecbc50ec0"
server_key_iv: "f6ab040fc9209a430a5e500240c96858"
server_key_version: "encryptionkey@0"
I20260619 14:42:46.628244 2493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:42:46.630790 2508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:46.632153 2493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:42:46.632295 2493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "6bdc477b990644489fce0187cd7b70ea"
format_stamp: "Formatted at 2026-06-19 14:42:46 on dist-test-slave-cr22"
server_key: "8ef67b283bfd184174a0dbeecbc50ec0"
server_key_iv: "f6ab040fc9209a430a5e500240c96858"
server_key_version: "encryptionkey@0"
I20260619 14:42:46.632411 2493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:46.642446 2493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:46.645606 2493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:46.645805 2493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:46.647787 2493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:46.647863 2493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:46.647935 2493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:46.647971 2493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:42:46 dist-test-slave-cr22 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:46.658403 2493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32995
I20260619 14:42:46.658461 2623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32995 every 8 connection(s)
I20260619 14:42:46.659552 2493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:42:46.662397 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.648628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51405 (local address 127.0.98.62:35575)
0619 14:42:46.648761 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:46.648768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:46.649646 (+ 878us) server_negotiation.cc:407] Connection header received
0619 14:42:46.650672 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:46.650676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:46.650735 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:46.650789 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:46.652570 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.653468 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.655263 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.655640 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.655766 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.658770 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:46.658774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:46.658777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:46.658808 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:46.660496 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:46.660970 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:46.660973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:46.660974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:46.661044 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:46.661383 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:46.661386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:46.661387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:46.661574 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:46.661657 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:46.662159 (+ 502us) server_negotiation.cc:299] Negotiation successful
0619 14:42:46.662256 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:42:46.663193 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.649033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35575 (local address 127.0.98.3:51405)
0619 14:42:46.649483 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:46.649531 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:46.650391 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:46.651234 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:46.651244 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:46.651763 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:46.652418 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.652438 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.653609 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.653614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.655142 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.655158 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.655806 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.655811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.655999 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.656658 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:46.656681 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:46.658585 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:46.660602 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:46.660607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:46.660619 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:46.660882 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:46.661124 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:46.661126 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:46.661127 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:46.661260 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:46.661671 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:46.661675 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:46.661947 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 14:42:46.662100 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 14:42:46.662335 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":135,"threads_started":1}
I20260619 14:42:46.664434 2624 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35575
I20260619 14:42:46.664570 2624 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:46.665104 2624 heartbeater.cc:511] Master 127.0.98.62:35575 requested a full tablet report, sending...
I20260619 14:42:46.666275 2137 ts_manager.cc:194] Registered new tserver with Master: 6bdc477b990644489fce0187cd7b70ea (127.0.98.3:32995)
I20260619 14:42:46.666918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 2493
I20260619 14:42:46.666989 2137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51405
I20260619 14:42:46.667035 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:42:46.667358 392 external_mini_cluster.cc:1428] Setting key a4dc510211d7326b5e8af1c4e1ef24ea
I20260619 14:42:46.669395 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:42:46.674757 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.667291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36737 (local address 127.0.98.62:35575)
0619 14:42:46.667386 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:46.667390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:46.667401 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:42:46.667475 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:46.667480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:46.667520 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:46.667595 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:46.668011 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.668734 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.669800 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.670030 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.670137 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.671294 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:46.671297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:46.671300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:46.671322 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:46.672827 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:46.673264 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:46.673269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:46.673271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:46.673358 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:46.673637 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:46.673640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:46.673642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:46.673819 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:46.673925 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:46.674447 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 14:42:46.674568 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 14:42:46.674778 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.666842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35575 (local address 127.0.98.3:36737)
0619 14:42:46.667113 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:46.667128 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:46.667232 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:46.667612 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:46.667614 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:46.667804 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:46.667896 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.667902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.668874 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.668878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.669686 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.669695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.670139 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.670143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.670230 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.670761 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:46.670772 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:46.671151 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:46.672943 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:46.672948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:46.672950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:46.673164 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:46.673448 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:46.673452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:46.673454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:46.673540 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:46.673914 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:46.673916 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:46.674222 (+ 306us) client_negotiation.cc:769] Sending connection context
0619 14:42:46.674387 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 14:42:46.674567 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":91,"threads_started":1}
Jun 19 14:42:46 dist-test-slave-cr22 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:42:46 dist-test-slave-cr22 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:46.698227 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.689169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33768 (local address 127.0.98.62:35575)
0619 14:42:46.689311 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:46.689315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:46.689356 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:42:46.689399 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:46.689402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:46.689454 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:46.689527 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:46.690234 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.690879 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.691912 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.692119 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.692204 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.694588 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:46.694592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:46.694594 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:46.694619 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:46.696527 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:46.697033 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:46.697036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:46.697037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:46.697110 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:46.697419 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:46.697422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:46.697423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:46.697596 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:46.697698 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:46.697996 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 14:42:46.698093 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:42:46.701481 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:42:46.704957 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:42:46.720515 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.714194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.0.98.1:45369)
0619 14:42:46.714799 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:46.714815 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:46.715156 (+ 341us) server_negotiation.cc:407] Connection header received
0619 14:42:46.715241 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:46.715247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:46.715486 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:46.715638 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:46.716420 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.717259 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.718828 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.719722 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.719841 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.720016 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:46.720240 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 14:42:46.720342 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":57,"threads_started":1}
I20260619 14:42:46.720541 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.714440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45007 (local address 127.0.0.1:50680)
0619 14:42:46.714944 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:46.714957 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:46.715020 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:46.715925 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:46.715928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:46.715947 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:46.716080 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.716090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.717835 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.717839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.718994 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.719002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.720001 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.720005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.720090 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.720109 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:42:46.720167 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 14:42:46.720356 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":58,"thread_start_us":87,"threads_started":1}
I20260619 14:42:46.720748 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.714149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32995 (local address 127.0.0.1:60070)
0619 14:42:46.714817 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:46.714873 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:46.715022 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:46.716133 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:46.716138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:46.716157 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:46.716344 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.716352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.717870 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.717875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.719025 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.719034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.720306 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.720309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.720396 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.720417 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:42:46.720485 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:42:46.720575 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":47,"spinlock_wait_cycles":12928,"thread_start_us":126,"threads_started":1}
I20260619 14:42:46.720789 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.714908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50680 (local address 127.0.98.2:45007)
0619 14:42:46.715357 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:46.715374 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:46.715399 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:42:46.715525 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:46.715531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:46.715737 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:46.716022 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:46.716163 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.717698 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.719132 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.719853 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.720307 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.720465 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:46.720518 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 14:42:46.720621 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":115,"threads_started":1}
I20260619 14:42:46.721242 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.715297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60070 (local address 127.0.98.3:32995)
0619 14:42:46.715707 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:46.715722 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:46.715743 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:42:46.715816 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:46.715822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:46.715982 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:46.716130 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:46.716488 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.717727 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.719156 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.719977 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.720093 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.720246 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:46.720955 (+ 709us) server_negotiation.cc:299] Negotiation successful
0619 14:42:46.721079 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":103,"threads_started":1}
I20260619 14:42:46.721909 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.714073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45369 (local address 127.0.0.1:47526)
0619 14:42:46.714816 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:46.714873 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:46.715020 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:46.715855 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:46.715861 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:46.715883 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:46.716303 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.716310 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.717371 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.717377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.718632 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.718642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.719864 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.719869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.720016 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.720064 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 14:42:46.720190 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 14:42:46.721775 (+ 1585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":40,"spinlock_wait_cycles":4992,"thread_start_us":95,"threads_started":1}
I20260619 14:42:46.723950 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 644b8626103247478127118653175e1d (DEFAULT_TABLE table=test-table [id=f4ad7bce7ca74606a79cc4f30f4fbed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:46.723950 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 644b8626103247478127118653175e1d (DEFAULT_TABLE table=test-table [id=f4ad7bce7ca74606a79cc4f30f4fbed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:46.724449 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 644b8626103247478127118653175e1d (DEFAULT_TABLE table=test-table [id=f4ad7bce7ca74606a79cc4f30f4fbed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:46.725083 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 644b8626103247478127118653175e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:46.725083 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 644b8626103247478127118653175e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:46.725416 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 644b8626103247478127118653175e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:46.732997 2647 tablet_bootstrap.cc:492] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239: Bootstrap starting.
I20260619 14:42:46.734452 2649 tablet_bootstrap.cc:492] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09: Bootstrap starting.
I20260619 14:42:46.734452 2648 tablet_bootstrap.cc:492] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea: Bootstrap starting.
I20260619 14:42:46.735304 2647 tablet_bootstrap.cc:654] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:46.736783 2648 tablet_bootstrap.cc:654] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:46.736876 2649 tablet_bootstrap.cc:654] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:46.736501 2647 log.cc:824] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239: Log is configured to *not* fsync() on all Append() calls
I20260619 14:42:46.737866 2649 log.cc:824] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09: Log is configured to *not* fsync() on all Append() calls
I20260619 14:42:46.737866 2648 log.cc:824] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea: Log is configured to *not* fsync() on all Append() calls
I20260619 14:42:46.739063 2647 tablet_bootstrap.cc:492] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239: No bootstrap required, opened a new log
I20260619 14:42:46.739277 2647 ts_tablet_manager.cc:1397] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:42:46.740025 2648 tablet_bootstrap.cc:492] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea: No bootstrap required, opened a new log
I20260619 14:42:46.740025 2649 tablet_bootstrap.cc:492] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09: No bootstrap required, opened a new log
I20260619 14:42:46.740218 2649 ts_tablet_manager.cc:1397] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260619 14:42:46.740221 2648 ts_tablet_manager.cc:1397] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:42:46.742643 2647 raft_consensus.cc:348] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } }
I20260619 14:42:46.742913 2647 raft_consensus.cc:374] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:46.742991 2647 raft_consensus.cc:729] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28125b7821d2490897d0b36297a2b239, State: Initialized, Role: FOLLOWER
I20260619 14:42:46.743840 2647 consensus_queue.cc:260] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } }
I20260619 14:42:46.744367 2649 raft_consensus.cc:348] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } }
I20260619 14:42:46.744367 2648 raft_consensus.cc:348] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } }
I20260619 14:42:46.744619 2648 raft_consensus.cc:374] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:46.744638 2344 heartbeater.cc:503] Master 127.0.98.62:35575 was elected leader, sending a full tablet report...
I20260619 14:42:46.744717 2648 raft_consensus.cc:729] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bdc477b990644489fce0187cd7b70ea, State: Initialized, Role: FOLLOWER
I20260619 14:42:46.744807 2649 raft_consensus.cc:374] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:46.744901 2649 raft_consensus.cc:729] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00177649bdaf493a8a03c6ebba1d2d09, State: Initialized, Role: FOLLOWER
I20260619 14:42:46.745095 2647 ts_tablet_manager.cc:1428] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:42:46.745445 2648 consensus_queue.cc:260] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } }
I20260619 14:42:46.745385 2649 consensus_queue.cc:260] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } }
I20260619 14:42:46.746145 2624 heartbeater.cc:503] Master 127.0.98.62:35575 was elected leader, sending a full tablet report...
I20260619 14:42:46.746390 2648 ts_tablet_manager.cc:1428] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:42:46.746953 2649 ts_tablet_manager.cc:1428] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:42:46.747556 2484 heartbeater.cc:503] Master 127.0.98.62:35575 was elected leader, sending a full tablet report...
I20260619 14:42:46.797315 2653 raft_consensus.cc:482] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:42:46.797533 2653 raft_consensus.cc:504] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } }
I20260619 14:42:46.798717 2653 leader_election.cc:288] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bdc477b990644489fce0187cd7b70ea (127.0.98.3:32995), 00177649bdaf493a8a03c6ebba1d2d09 (127.0.98.2:45007)
I20260619 14:42:46.802980 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.799029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32995 (local address 127.0.98.1:49475)
0619 14:42:46.799372 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:46.799387 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:46.799523 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:46.799832 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:46.799835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:46.799853 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:46.799982 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.799991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.800869 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.800873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.801715 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.801724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.802423 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.802427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.802520 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.802555 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 14:42:46.802721 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 14:42:46.802818 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":108,"threads_started":1}
I20260619 14:42:46.802980 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.799136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49475 (local address 127.0.98.3:32995)
0619 14:42:46.799276 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:46.799282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:46.799500 (+ 218us) server_negotiation.cc:407] Connection header received
0619 14:42:46.799643 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:46.799646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:46.799705 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:46.799789 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:46.800111 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.800751 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.801845 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.802308 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.802388 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.802441 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:46.802685 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 14:42:46.802781 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:42:46.803287 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.799077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45007 (local address 127.0.98.1:44281)
0619 14:42:46.799524 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:46.799539 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:46.799668 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:46.800020 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:46.800022 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:46.800035 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:46.800178 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.800185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.801002 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.801005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.801869 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.801878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.802840 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.802844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.802940 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.802963 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:42:46.803052 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 14:42:46.803120 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":119,"threads_started":1}
I20260619 14:42:46.803318 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.799257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44281 (local address 127.0.98.2:45007)
0619 14:42:46.799413 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:46.799419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:46.799732 (+ 313us) server_negotiation.cc:407] Connection header received
0619 14:42:46.799853 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:46.799856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:46.799907 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:46.799999 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:46.800288 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.800893 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.802120 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.802705 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.802807 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.802863 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:46.803083 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 14:42:46.803187 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:42:46.803807 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "644b8626103247478127118653175e1d" candidate_uuid: "28125b7821d2490897d0b36297a2b239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bdc477b990644489fce0187cd7b70ea" is_pre_election: true
I20260619 14:42:46.804045 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "644b8626103247478127118653175e1d" candidate_uuid: "28125b7821d2490897d0b36297a2b239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00177649bdaf493a8a03c6ebba1d2d09" is_pre_election: true
I20260619 14:42:46.804114 2578 raft_consensus.cc:2454] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28125b7821d2490897d0b36297a2b239 in term 0.
I20260619 14:42:46.804329 2438 raft_consensus.cc:2454] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28125b7821d2490897d0b36297a2b239 in term 0.
I20260619 14:42:46.804683 2231 leader_election.cc:302] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28125b7821d2490897d0b36297a2b239, 6bdc477b990644489fce0187cd7b70ea; no voters:
I20260619 14:42:46.805008 2653 raft_consensus.cc:2781] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:42:46.805088 2653 raft_consensus.cc:482] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:42:46.805120 2653 raft_consensus.cc:3037] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:46.806113 2653 raft_consensus.cc:504] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } }
I20260619 14:42:46.806437 2653 leader_election.cc:288] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [CANDIDATE]: Term 1 election: Requested vote from peers 6bdc477b990644489fce0187cd7b70ea (127.0.98.3:32995), 00177649bdaf493a8a03c6ebba1d2d09 (127.0.98.2:45007)
I20260619 14:42:46.806778 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "644b8626103247478127118653175e1d" candidate_uuid: "28125b7821d2490897d0b36297a2b239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bdc477b990644489fce0187cd7b70ea"
I20260619 14:42:46.806797 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "644b8626103247478127118653175e1d" candidate_uuid: "28125b7821d2490897d0b36297a2b239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00177649bdaf493a8a03c6ebba1d2d09"
I20260619 14:42:46.806905 2578 raft_consensus.cc:3037] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:46.807011 2438 raft_consensus.cc:3037] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:46.807991 2578 raft_consensus.cc:2454] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28125b7821d2490897d0b36297a2b239 in term 1.
I20260619 14:42:46.808282 2231 leader_election.cc:302] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28125b7821d2490897d0b36297a2b239, 6bdc477b990644489fce0187cd7b70ea; no voters:
I20260619 14:42:46.808291 2438 raft_consensus.cc:2454] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28125b7821d2490897d0b36297a2b239 in term 1.
I20260619 14:42:46.808454 2653 raft_consensus.cc:2781] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:42:46.808662 2653 raft_consensus.cc:686] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [term 1 LEADER]: Becoming Leader. State: Replica: 28125b7821d2490897d0b36297a2b239, State: Running, Role: LEADER
I20260619 14:42:46.809015 2653 consensus_queue.cc:237] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } }
I20260619 14:42:46.812707 2137 catalog_manager.cc:5495] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28125b7821d2490897d0b36297a2b239 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "28125b7821d2490897d0b36297a2b239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:42:46 dist-test-slave-cr22 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:46.861896 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.853231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.0.98.62:35575)
0619 14:42:46.853494 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:46.853500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:46.853707 (+ 207us) server_negotiation.cc:407] Connection header received
0619 14:42:46.853942 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:46.853946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:46.853993 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:46.854083 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:46.854726 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.855280 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.856116 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.856323 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.856437 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.858644 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:46.858648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:46.858652 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:46.858677 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:46.860159 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:46.860668 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:46.860671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:46.860673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:46.860746 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:46.861020 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:46.861023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:46.861024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:46.861202 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:46.861309 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:46.861584 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 14:42:46.861747 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":49}
I20260619 14:42:46.861896 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:46.853124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35575 (local address 127.0.0.1:33776)
0619 14:42:46.853613 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:46.853627 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:46.853775 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:46.854098 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:46.854101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:46.854519 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:46.854609 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.854616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.855395 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.855401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.855999 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:46.856006 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:46.856448 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:46.856451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:46.856587 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:46.857123 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:46.857135 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:46.858508 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:46.860301 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:46.860311 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:46.860318 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:46.860552 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:46.860836 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:46.860839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:46.860844 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:46.860925 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:46.861309 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:46.861317 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:46.861426 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 14:42:46.861544 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 14:42:46.861697 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":39,"thread_start_us":133,"threads_started":1}
I20260619 14:42:46.865801 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33776:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:42:46.866575 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:42:46.871767 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 97c4cbc1b3e144fea1b1257328798ae2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=80d1b48fbd53495682cbf126a788cdbc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:42:46.871930 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 97c4cbc1b3e144fea1b1257328798ae2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=80d1b48fbd53495682cbf126a788cdbc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:42:46.872556 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c4cbc1b3e144fea1b1257328798ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:46.872705 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c4cbc1b3e144fea1b1257328798ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:46.875298 2647 tablet_bootstrap.cc:492] T 97c4cbc1b3e144fea1b1257328798ae2 P 28125b7821d2490897d0b36297a2b239: Bootstrap starting.
I20260619 14:42:46.875326 2649 tablet_bootstrap.cc:492] T 97c4cbc1b3e144fea1b1257328798ae2 P 00177649bdaf493a8a03c6ebba1d2d09: Bootstrap starting.
I20260619 14:42:46.876257 2649 tablet_bootstrap.cc:654] T 97c4cbc1b3e144fea1b1257328798ae2 P 00177649bdaf493a8a03c6ebba1d2d09: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:46.876340 2647 tablet_bootstrap.cc:654] T 97c4cbc1b3e144fea1b1257328798ae2 P 28125b7821d2490897d0b36297a2b239: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:46.875923 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 97c4cbc1b3e144fea1b1257328798ae2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=80d1b48fbd53495682cbf126a788cdbc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:42:46.876678 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c4cbc1b3e144fea1b1257328798ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:46.877740 2649 tablet_bootstrap.cc:492] T 97c4cbc1b3e144fea1b1257328798ae2 P 00177649bdaf493a8a03c6ebba1d2d09: No bootstrap required, opened a new log
I20260619 14:42:46.877849 2649 ts_tablet_manager.cc:1397] T 97c4cbc1b3e144fea1b1257328798ae2 P 00177649bdaf493a8a03c6ebba1d2d09: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:42:46.878341 2649 raft_consensus.cc:348] T 97c4cbc1b3e144fea1b1257328798ae2 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:46.878460 2649 raft_consensus.cc:374] T 97c4cbc1b3e144fea1b1257328798ae2 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:46.878492 2649 raft_consensus.cc:729] T 97c4cbc1b3e144fea1b1257328798ae2 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00177649bdaf493a8a03c6ebba1d2d09, State: Initialized, Role: FOLLOWER
I20260619 14:42:46.878625 2649 consensus_queue.cc:260] T 97c4cbc1b3e144fea1b1257328798ae2 P 00177649bdaf493a8a03c6ebba1d2d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:46.878891 2649 ts_tablet_manager.cc:1428] T 97c4cbc1b3e144fea1b1257328798ae2 P 00177649bdaf493a8a03c6ebba1d2d09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:42:46.878981 2655 tablet_replica.cc:439] T 97c4cbc1b3e144fea1b1257328798ae2 P 00177649bdaf493a8a03c6ebba1d2d09: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } } }
I20260619 14:42:46.879138 2655 tablet_replica.cc:442] T 97c4cbc1b3e144fea1b1257328798ae2 P 00177649bdaf493a8a03c6ebba1d2d09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:42:46.879398 2647 tablet_bootstrap.cc:492] T 97c4cbc1b3e144fea1b1257328798ae2 P 28125b7821d2490897d0b36297a2b239: No bootstrap required, opened a new log
I20260619 14:42:46.879519 2647 ts_tablet_manager.cc:1397] T 97c4cbc1b3e144fea1b1257328798ae2 P 28125b7821d2490897d0b36297a2b239: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 14:42:46.879757 2648 tablet_bootstrap.cc:492] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea: Bootstrap starting.
I20260619 14:42:46.879985 2647 raft_consensus.cc:348] T 97c4cbc1b3e144fea1b1257328798ae2 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:46.880093 2647 raft_consensus.cc:374] T 97c4cbc1b3e144fea1b1257328798ae2 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:46.880125 2647 raft_consensus.cc:729] T 97c4cbc1b3e144fea1b1257328798ae2 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28125b7821d2490897d0b36297a2b239, State: Initialized, Role: FOLLOWER
I20260619 14:42:46.880287 2647 consensus_queue.cc:260] T 97c4cbc1b3e144fea1b1257328798ae2 P 28125b7821d2490897d0b36297a2b239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:46.880517 2653 tablet_replica.cc:439] T 97c4cbc1b3e144fea1b1257328798ae2 P 28125b7821d2490897d0b36297a2b239: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } } }
I20260619 14:42:46.880646 2653 tablet_replica.cc:442] T 97c4cbc1b3e144fea1b1257328798ae2 P 28125b7821d2490897d0b36297a2b239: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:42:46.880868 2648 tablet_bootstrap.cc:654] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:46.880931 2647 ts_tablet_manager.cc:1428] T 97c4cbc1b3e144fea1b1257328798ae2 P 28125b7821d2490897d0b36297a2b239: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:42:46.882447 2648 tablet_bootstrap.cc:492] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea: No bootstrap required, opened a new log
I20260619 14:42:46.882562 2648 ts_tablet_manager.cc:1397] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:42:46.883114 2648 raft_consensus.cc:348] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:46.883200 2648 raft_consensus.cc:374] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:46.883220 2648 raft_consensus.cc:729] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bdc477b990644489fce0187cd7b70ea, State: Initialized, Role: FOLLOWER
I20260619 14:42:46.883351 2648 consensus_queue.cc:260] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:46.883628 2648 ts_tablet_manager.cc:1428] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:42:46.883652 2654 tablet_replica.cc:439] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } } }
I20260619 14:42:46.883832 2654 tablet_replica.cc:442] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 14:42:46.910928 2625 tablet.cc:2369] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:42:46.946264 2485 tablet.cc:2369] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:42:46.978458 2345 tablet.cc:2369] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:42:47.253120 2654 raft_consensus.cc:482] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:42:47.253274 2654 raft_consensus.cc:504] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:47.254532 2654 leader_election.cc:288] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28125b7821d2490897d0b36297a2b239 (127.0.98.1:45369), 00177649bdaf493a8a03c6ebba1d2d09 (127.0.98.2:45007)
I20260619 14:42:47.260329 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:47.255006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45007 (local address 127.0.98.3:49623)
0619 14:42:47.255276 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:47.255288 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:47.255374 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:47.255839 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:47.255842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:47.255876 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:47.255959 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:47.255966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.257250 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.257254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:47.258792 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:47.258807 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.259751 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.259755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:47.259920 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:47.259939 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:42:47.260009 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 14:42:47.260114 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":218,"spinlock_wait_cycles":3072,"thread_start_us":76,"threads_started":1}
I20260619 14:42:47.260346 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:47.255077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49623 (local address 127.0.98.2:45007)
0619 14:42:47.255207 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:47.255212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:47.255429 (+ 217us) server_negotiation.cc:407] Connection header received
0619 14:42:47.255526 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:47.255528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:47.255574 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:47.255667 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:47.256152 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.257104 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.258954 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.259584 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.259704 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:47.259773 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:47.260075 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 14:42:47.260236 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:42:47.260347 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:47.254844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38729 (local address 127.0.98.1:45369)
0619 14:42:47.255107 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:47.255123 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:47.255277 (+ 154us) server_negotiation.cc:407] Connection header received
0619 14:42:47.255514 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:47.255517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:47.255572 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:47.255668 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:42:47.256204 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.257083 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.258954 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.259623 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.259719 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:47.259854 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:47.260075 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 14:42:47.260181 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":56,"threads_started":1}
I20260619 14:42:47.260362 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:47.254757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45369 (local address 127.0.98.3:38729)
0619 14:42:47.255155 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:47.255172 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:47.255310 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:47.255780 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:47.255784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:47.255810 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:42:47.255959 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:47.255966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.257250 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.257254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:47.258792 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:47.258804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.259751 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.259755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:47.259883 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:47.259912 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:42:47.260008 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 14:42:47.260114 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":122,"spinlock_wait_cycles":24960,"thread_start_us":77,"threads_started":1}
I20260619 14:42:47.260856 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c4cbc1b3e144fea1b1257328798ae2" candidate_uuid: "6bdc477b990644489fce0187cd7b70ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00177649bdaf493a8a03c6ebba1d2d09" is_pre_election: true
I20260619 14:42:47.261023 2438 raft_consensus.cc:2454] T 97c4cbc1b3e144fea1b1257328798ae2 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bdc477b990644489fce0187cd7b70ea in term 0.
I20260619 14:42:47.260974 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c4cbc1b3e144fea1b1257328798ae2" candidate_uuid: "6bdc477b990644489fce0187cd7b70ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28125b7821d2490897d0b36297a2b239" is_pre_election: true
I20260619 14:42:47.261191 2298 raft_consensus.cc:2454] T 97c4cbc1b3e144fea1b1257328798ae2 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bdc477b990644489fce0187cd7b70ea in term 0.
I20260619 14:42:47.261422 2512 leader_election.cc:302] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00177649bdaf493a8a03c6ebba1d2d09, 6bdc477b990644489fce0187cd7b70ea; no voters:
I20260619 14:42:47.261873 2654 raft_consensus.cc:2781] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:42:47.261932 2654 raft_consensus.cc:482] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:42:47.261961 2654 raft_consensus.cc:3037] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:47.262964 2654 raft_consensus.cc:504] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:47.263355 2654 leader_election.cc:288] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea [CANDIDATE]: Term 1 election: Requested vote from peers 28125b7821d2490897d0b36297a2b239 (127.0.98.1:45369), 00177649bdaf493a8a03c6ebba1d2d09 (127.0.98.2:45007)
I20260619 14:42:47.263720 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c4cbc1b3e144fea1b1257328798ae2" candidate_uuid: "6bdc477b990644489fce0187cd7b70ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28125b7821d2490897d0b36297a2b239"
I20260619 14:42:47.263762 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c4cbc1b3e144fea1b1257328798ae2" candidate_uuid: "6bdc477b990644489fce0187cd7b70ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00177649bdaf493a8a03c6ebba1d2d09"
I20260619 14:42:47.263847 2298 raft_consensus.cc:3037] T 97c4cbc1b3e144fea1b1257328798ae2 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:47.263875 2438 raft_consensus.cc:3037] T 97c4cbc1b3e144fea1b1257328798ae2 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:47.265021 2438 raft_consensus.cc:2454] T 97c4cbc1b3e144fea1b1257328798ae2 P 00177649bdaf493a8a03c6ebba1d2d09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bdc477b990644489fce0187cd7b70ea in term 1.
I20260619 14:42:47.265021 2298 raft_consensus.cc:2454] T 97c4cbc1b3e144fea1b1257328798ae2 P 28125b7821d2490897d0b36297a2b239 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bdc477b990644489fce0187cd7b70ea in term 1.
I20260619 14:42:47.265372 2509 leader_election.cc:302] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28125b7821d2490897d0b36297a2b239, 6bdc477b990644489fce0187cd7b70ea; no voters:
I20260619 14:42:47.265539 2654 raft_consensus.cc:2781] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:42:47.265733 2654 raft_consensus.cc:686] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea [term 1 LEADER]: Becoming Leader. State: Replica: 6bdc477b990644489fce0187cd7b70ea, State: Running, Role: LEADER
I20260619 14:42:47.266099 2654 consensus_queue.cc:237] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:47.268064 2672 tablet_replica.cc:439] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea: TxnStatusTablet state changed. Reason: New leader 6bdc477b990644489fce0187cd7b70ea. Latest consensus state: current_term: 1 leader_uuid: "6bdc477b990644489fce0187cd7b70ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } } }
I20260619 14:42:47.268178 2672 tablet_replica.cc:442] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:42:47.268440 2674 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:42:47.268654 2674 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:42:47.269466 2135 catalog_manager.cc:5495] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bdc477b990644489fce0187cd7b70ea (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6bdc477b990644489fce0187cd7b70ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } health_report { overall_health: HEALTHY } } }
I20260619 14:42:47.328655 2653 consensus_queue.cc:1035] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:42:47.334860 2679 consensus_queue.cc:1035] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:42:47.335340 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:47.329445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32995 (local address 127.0.0.1:60072)
0619 14:42:47.329677 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:47.329693 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:47.329809 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:47.330333 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:47.330336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:47.330345 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:42:47.330455 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:47.330461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.332109 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.332112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:47.333022 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:47.333034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.333615 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.333619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:47.333706 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:47.333776 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:42:47.335007 (+ 1231us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:42:47.335025 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:42:47.335069 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 14:42:47.335162 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260619 14:42:47.335846 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:47.329523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60072 (local address 127.0.98.3:32995)
0619 14:42:47.330022 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:47.330026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:47.330038 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:42:47.330111 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:47.330114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:47.330186 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:47.330306 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:42:47.330571 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.331986 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.333155 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.333483 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.334314 (+ 831us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:47.334393 (+ 79us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:42:47.334893 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:42:47.335575 (+ 682us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:47.335610 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 14:42:47.335660 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":58,"threads_started":1}
I20260619 14:42:47.340728 2437 raft_consensus.cc:1261] T 97c4cbc1b3e144fea1b1257328798ae2 P 00177649bdaf493a8a03c6ebba1d2d09 [term 1 FOLLOWER]: Refusing update from remote peer 6bdc477b990644489fce0187cd7b70ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:42:47.341145 2298 raft_consensus.cc:1261] T 97c4cbc1b3e144fea1b1257328798ae2 P 28125b7821d2490897d0b36297a2b239 [term 1 FOLLOWER]: Refusing update from remote peer 6bdc477b990644489fce0187cd7b70ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:42:47.341603 2654 consensus_queue.cc:1035] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:42:47.341804 2672 consensus_queue.cc:1035] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:42:47.344641 2655 tablet_replica.cc:439] T 97c4cbc1b3e144fea1b1257328798ae2 P 00177649bdaf493a8a03c6ebba1d2d09: TxnStatusTablet state changed. Reason: New leader 6bdc477b990644489fce0187cd7b70ea. Latest consensus state: current_term: 1 leader_uuid: "6bdc477b990644489fce0187cd7b70ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } } }
I20260619 14:42:47.344765 2655 tablet_replica.cc:442] T 97c4cbc1b3e144fea1b1257328798ae2 P 00177649bdaf493a8a03c6ebba1d2d09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:42:47.346482 2679 tablet_replica.cc:439] T 97c4cbc1b3e144fea1b1257328798ae2 P 28125b7821d2490897d0b36297a2b239: TxnStatusTablet state changed. Reason: New leader 6bdc477b990644489fce0187cd7b70ea. Latest consensus state: current_term: 1 leader_uuid: "6bdc477b990644489fce0187cd7b70ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } } }
I20260619 14:42:47.346601 2679 tablet_replica.cc:442] T 97c4cbc1b3e144fea1b1257328798ae2 P 28125b7821d2490897d0b36297a2b239: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:42:47.349337 2654 tablet_replica.cc:439] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6bdc477b990644489fce0187cd7b70ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } } }
I20260619 14:42:47.349473 2654 tablet_replica.cc:442] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:42:47.349432 2672 tablet_replica.cc:439] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6bdc477b990644489fce0187cd7b70ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } } }
I20260619 14:42:47.349505 2672 tablet_replica.cc:442] T 97c4cbc1b3e144fea1b1257328798ae2 P 6bdc477b990644489fce0187cd7b70ea: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:42:47.350293 2655 tablet_replica.cc:439] T 97c4cbc1b3e144fea1b1257328798ae2 P 00177649bdaf493a8a03c6ebba1d2d09: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6bdc477b990644489fce0187cd7b70ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } } }
I20260619 14:42:47.350406 2655 tablet_replica.cc:442] T 97c4cbc1b3e144fea1b1257328798ae2 P 00177649bdaf493a8a03c6ebba1d2d09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:42:47.350433 2681 mvcc.cc:205] Tried to move back new op lower bound from 7298581165422632960 to 7298581165124673536. Current Snapshot: MvccSnapshot[applied={T|T < 7298581165422632960}]
I20260619 14:42:47.350519 2680 mvcc.cc:205] Tried to move back new op lower bound from 7298581165422632960 to 7298581165124673536. Current Snapshot: MvccSnapshot[applied={T|T < 7298581165422632960}]
I20260619 14:42:47.350708 2679 tablet_replica.cc:439] T 97c4cbc1b3e144fea1b1257328798ae2 P 28125b7821d2490897d0b36297a2b239: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6bdc477b990644489fce0187cd7b70ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } } }
I20260619 14:42:47.350805 2679 tablet_replica.cc:442] T 97c4cbc1b3e144fea1b1257328798ae2 P 28125b7821d2490897d0b36297a2b239: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:42:47.361505 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:47.357435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47536 (local address 127.0.98.1:45369)
0619 14:42:47.357550 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:47.357554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:47.357698 (+ 144us) server_negotiation.cc:407] Connection header received
0619 14:42:47.357827 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:47.357830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:47.357887 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:47.357989 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:42:47.358204 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.359013 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.359993 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.360222 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.360340 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:47.360753 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:42:47.361133 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:42:47.361269 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:47.361310 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 14:42:47.361362 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:42:47.373472 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:47.369240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32995 (local address 127.0.98.1:48267)
0619 14:42:47.369620 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:47.369638 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:47.369750 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:47.369996 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:47.369998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:47.370004 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:42:47.370102 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:47.370107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.371179 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.371183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:47.372024 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:47.372035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.372618 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.372621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:47.372775 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:47.372819 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:42:47.373179 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:42:47.373187 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:42:47.373257 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 14:42:47.373334 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260619 14:42:47.373469 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:47.369385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48267 (local address 127.0.98.3:32995)
0619 14:42:47.369529 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:47.369534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:47.369721 (+ 187us) server_negotiation.cc:407] Connection header received
0619 14:42:47.369830 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:47.369834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:47.369887 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:47.369973 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:42:47.370261 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.371029 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.372144 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.372499 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.372612 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:47.372979 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:42:47.373062 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:42:47.373140 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:47.373286 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 14:42:47.373347 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:42:47.401846 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:47.397931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41565 (local address 127.0.98.1:45369)
0619 14:42:47.398053 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:47.398057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:47.398297 (+ 240us) server_negotiation.cc:407] Connection header received
0619 14:42:47.398456 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:47.398459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:47.398505 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:47.398596 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:42:47.398885 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.399595 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.400705 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.400995 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.401106 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:47.401394 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:42:47.401474 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:42:47.401549 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:47.401647 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 14:42:47.401695 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:42:47.401856 2697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:47.397883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45369 (local address 127.0.98.3:41565)
0619 14:42:47.398207 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:47.398220 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:47.398350 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:47.398606 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:47.398609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:47.398616 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:42:47.398777 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:47.398783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.399724 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.399728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:47.400580 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:47.400590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:47.401106 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:47.401110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:47.401204 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:47.401261 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:42:47.401563 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:42:47.401571 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:42:47.401628 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 14:42:47.401707 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":64,"threads_started":1}
I20260619 14:42:47.428968 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:42:47.429160 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:42:47.431581 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb: Sending DeleteTablet for 3 replicas of tablet 644b8626103247478127118653175e1d
I20260619 14:42:47.432700 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet 644b8626103247478127118653175e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:42:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50680
I20260619 14:42:47.433152 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 14:42:47.433281 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet 644b8626103247478127118653175e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:42:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47526
W20260619 14:42:47.433545 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:42:47.433900 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet 644b8626103247478127118653175e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:42:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60070
I20260619 14:42:47.434446 2700 tablet_replica.cc:330] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09: stopping tablet replica
I20260619 14:42:47.434921 2700 raft_consensus.cc:2229] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:42:47.435206 2700 raft_consensus.cc:2258] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:42:47.435689 2701 tablet_replica.cc:330] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239: stopping tablet replica
I20260619 14:42:47.436496 2702 tablet_replica.cc:330] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea: stopping tablet replica
I20260619 14:42:47.436857 2701 raft_consensus.cc:2229] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:42:47.437220 2702 raft_consensus.cc:2229] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:42:47.438083 2700 ts_tablet_manager.cc:1905] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:42:47.438308 2702 raft_consensus.cc:2258] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:42:47.438720 2701 raft_consensus.cc:2258] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:42:47.438884 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 59536dfd577c4c199bc92891160642f9 (DEFAULT_TABLE table=test-table [id=b96dd1106870497581c0735943b4a6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:47.439179 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59536dfd577c4c199bc92891160642f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:47.440056 2702 ts_tablet_manager.cc:1905] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:42:47.441283 2701 ts_tablet_manager.cc:1905] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:42:47.443050 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 59536dfd577c4c199bc92891160642f9 (DEFAULT_TABLE table=test-table [id=b96dd1106870497581c0735943b4a6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:47.443320 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59536dfd577c4c199bc92891160642f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:47.444542 2703 tablet_bootstrap.cc:492] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea: Bootstrap starting.
I20260619 14:42:47.442507 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 59536dfd577c4c199bc92891160642f9 (DEFAULT_TABLE table=test-table [id=b96dd1106870497581c0735943b4a6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:47.444855 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59536dfd577c4c199bc92891160642f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:47.445429 2703 tablet_bootstrap.cc:654] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:47.446516 2703 tablet_bootstrap.cc:492] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea: No bootstrap required, opened a new log
I20260619 14:42:47.446590 2703 ts_tablet_manager.cc:1397] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:42:47.446969 2703 raft_consensus.cc:348] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:47.447072 2703 raft_consensus.cc:374] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:47.447103 2703 raft_consensus.cc:729] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bdc477b990644489fce0187cd7b70ea, State: Initialized, Role: FOLLOWER
I20260619 14:42:47.447224 2703 consensus_queue.cc:260] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:47.447388 2705 tablet_bootstrap.cc:492] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239: Bootstrap starting.
I20260619 14:42:47.447540 2703 ts_tablet_manager.cc:1428] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:42:47.449229 2705 tablet_bootstrap.cc:654] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:47.450474 2705 tablet_bootstrap.cc:492] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239: No bootstrap required, opened a new log
I20260619 14:42:47.450564 2705 ts_tablet_manager.cc:1397] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:42:47.451020 2705 raft_consensus.cc:348] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:47.451143 2705 raft_consensus.cc:374] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:47.451182 2705 raft_consensus.cc:729] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28125b7821d2490897d0b36297a2b239, State: Initialized, Role: FOLLOWER
I20260619 14:42:47.451324 2705 consensus_queue.cc:260] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:47.451633 2705 ts_tablet_manager.cc:1428] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:42:47.451905 2702 ts_tablet_manager.cc:1918] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:42:47.452004 2702 log.cc:1196] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/644b8626103247478127118653175e1d
I20260619 14:42:47.452309 2702 ts_tablet_manager.cc:1939] T 644b8626103247478127118653175e1d P 6bdc477b990644489fce0187cd7b70ea: Deleting consensus metadata
I20260619 14:42:47.452943 2700 ts_tablet_manager.cc:1918] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:42:47.453110 2700 log.cc:1196] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/644b8626103247478127118653175e1d
I20260619 14:42:47.453475 2700 ts_tablet_manager.cc:1939] T 644b8626103247478127118653175e1d P 00177649bdaf493a8a03c6ebba1d2d09: Deleting consensus metadata
I20260619 14:42:47.453639 2127 catalog_manager.cc:4841] TS 6bdc477b990644489fce0187cd7b70ea (127.0.98.3:32995): tablet 644b8626103247478127118653175e1d (table test-table [id=f4ad7bce7ca74606a79cc4f30f4fbed7]) successfully deleted
I20260619 14:42:47.454648 2701 ts_tablet_manager.cc:1918] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:42:47.454766 2701 log.cc:1196] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/wal/wals/644b8626103247478127118653175e1d
I20260619 14:42:47.454912 2128 catalog_manager.cc:4841] TS 00177649bdaf493a8a03c6ebba1d2d09 (127.0.98.2:45007): tablet 644b8626103247478127118653175e1d (table test-table [id=f4ad7bce7ca74606a79cc4f30f4fbed7]) successfully deleted
I20260619 14:42:47.455063 2701 ts_tablet_manager.cc:1939] T 644b8626103247478127118653175e1d P 28125b7821d2490897d0b36297a2b239: Deleting consensus metadata
I20260619 14:42:47.455295 2707 tablet_bootstrap.cc:492] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09: Bootstrap starting.
I20260619 14:42:47.456069 2125 catalog_manager.cc:4841] TS 28125b7821d2490897d0b36297a2b239 (127.0.98.1:45369): tablet 644b8626103247478127118653175e1d (table test-table [id=f4ad7bce7ca74606a79cc4f30f4fbed7]) successfully deleted
I20260619 14:42:47.456326 2707 tablet_bootstrap.cc:654] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:47.457544 2707 tablet_bootstrap.cc:492] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09: No bootstrap required, opened a new log
I20260619 14:42:47.457634 2707 ts_tablet_manager.cc:1397] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:42:47.458065 2707 raft_consensus.cc:348] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:47.458185 2707 raft_consensus.cc:374] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:47.458225 2707 raft_consensus.cc:729] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00177649bdaf493a8a03c6ebba1d2d09, State: Initialized, Role: FOLLOWER
I20260619 14:42:47.458356 2707 consensus_queue.cc:260] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:47.458601 2707 ts_tablet_manager.cc:1428] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:42:47.511369 2695 raft_consensus.cc:482] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:42:47.511574 2695 raft_consensus.cc:504] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:47.512140 2695 leader_election.cc:288] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00177649bdaf493a8a03c6ebba1d2d09 (127.0.98.2:45007), 6bdc477b990644489fce0187cd7b70ea (127.0.98.3:32995)
I20260619 14:42:47.512607 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59536dfd577c4c199bc92891160642f9" candidate_uuid: "28125b7821d2490897d0b36297a2b239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00177649bdaf493a8a03c6ebba1d2d09" is_pre_election: true
I20260619 14:42:47.512662 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59536dfd577c4c199bc92891160642f9" candidate_uuid: "28125b7821d2490897d0b36297a2b239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bdc477b990644489fce0187cd7b70ea" is_pre_election: true
I20260619 14:42:47.512776 2438 raft_consensus.cc:2454] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28125b7821d2490897d0b36297a2b239 in term 0.
I20260619 14:42:47.512794 2578 raft_consensus.cc:2454] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28125b7821d2490897d0b36297a2b239 in term 0.
I20260619 14:42:47.513142 2231 leader_election.cc:302] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00177649bdaf493a8a03c6ebba1d2d09, 28125b7821d2490897d0b36297a2b239, 6bdc477b990644489fce0187cd7b70ea; no voters:
I20260619 14:42:47.513386 2695 raft_consensus.cc:2781] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:42:47.513466 2695 raft_consensus.cc:482] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:42:47.513520 2695 raft_consensus.cc:3037] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:47.514545 2695 raft_consensus.cc:504] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:47.514941 2695 leader_election.cc:288] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [CANDIDATE]: Term 1 election: Requested vote from peers 00177649bdaf493a8a03c6ebba1d2d09 (127.0.98.2:45007), 6bdc477b990644489fce0187cd7b70ea (127.0.98.3:32995)
I20260619 14:42:47.515336 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59536dfd577c4c199bc92891160642f9" candidate_uuid: "28125b7821d2490897d0b36297a2b239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bdc477b990644489fce0187cd7b70ea"
I20260619 14:42:47.515327 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59536dfd577c4c199bc92891160642f9" candidate_uuid: "28125b7821d2490897d0b36297a2b239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00177649bdaf493a8a03c6ebba1d2d09"
I20260619 14:42:47.515493 2438 raft_consensus.cc:3037] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:47.515493 2578 raft_consensus.cc:3037] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:47.516741 2578 raft_consensus.cc:2454] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28125b7821d2490897d0b36297a2b239 in term 1.
I20260619 14:42:47.516737 2438 raft_consensus.cc:2454] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28125b7821d2490897d0b36297a2b239 in term 1.
I20260619 14:42:47.517109 2232 leader_election.cc:302] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00177649bdaf493a8a03c6ebba1d2d09, 28125b7821d2490897d0b36297a2b239, 6bdc477b990644489fce0187cd7b70ea; no voters:
I20260619 14:42:47.517333 2695 raft_consensus.cc:2781] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:42:47.517457 2695 raft_consensus.cc:686] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [term 1 LEADER]: Becoming Leader. State: Replica: 28125b7821d2490897d0b36297a2b239, State: Running, Role: LEADER
I20260619 14:42:47.517601 2695 consensus_queue.cc:237] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:47.519721 2137 catalog_manager.cc:5495] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28125b7821d2490897d0b36297a2b239 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "28125b7821d2490897d0b36297a2b239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } health_report { overall_health: UNKNOWN } } }
I20260619 14:42:47.555953 2438 raft_consensus.cc:1261] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09 [term 1 FOLLOWER]: Refusing update from remote peer 28125b7821d2490897d0b36297a2b239: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:42:47.556022 2578 raft_consensus.cc:1261] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea [term 1 FOLLOWER]: Refusing update from remote peer 28125b7821d2490897d0b36297a2b239: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:42:47.556439 2679 consensus_queue.cc:1035] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:42:47.556689 2695 consensus_queue.cc:1035] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:42:47.561455 2683 mvcc.cc:205] Tried to move back new op lower bound from 7298581166304964608 to 7298581166153592832. Current Snapshot: MvccSnapshot[applied={T|T < 7298581166304964608}]
W20260619 14:42:47.574402 2680 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 14:42:49.131225 2135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:42:49.131373 2135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:42:49.134018 2135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb: Sending DeleteTablet for 3 replicas of tablet 59536dfd577c4c199bc92891160642f9
I20260619 14:42:49.134589 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet 59536dfd577c4c199bc92891160642f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:42:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47526
I20260619 14:42:49.134732 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet 59536dfd577c4c199bc92891160642f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:42:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60070
I20260619 14:42:49.134763 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet 59536dfd577c4c199bc92891160642f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:42:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50680
I20260619 14:42:49.135032 2735 tablet_replica.cc:330] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea: stopping tablet replica
I20260619 14:42:49.135136 2734 tablet_replica.cc:330] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239: stopping tablet replica
I20260619 14:42:49.135172 2736 tablet_replica.cc:330] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09: stopping tablet replica
I20260619 14:42:49.135185 2735 raft_consensus.cc:2229] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:42:49.135241 2734 raft_consensus.cc:2229] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:42:49.135286 2736 raft_consensus.cc:2229] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:42:49.135298 2735 raft_consensus.cc:2258] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:42:49.135392 2736 raft_consensus.cc:2258] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:42:49.135417 2734 raft_consensus.cc:2258] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:42:49.136112 2736 ts_tablet_manager.cc:1905] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:42:49.136229 2735 ts_tablet_manager.cc:1905] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:42:49.138847 2735 ts_tablet_manager.cc:1918] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:42:49.138847 2736 ts_tablet_manager.cc:1918] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:42:49.138931 2736 log.cc:1196] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/59536dfd577c4c199bc92891160642f9
I20260619 14:42:49.138929 2735 log.cc:1196] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/59536dfd577c4c199bc92891160642f9
I20260619 14:42:49.139209 2735 ts_tablet_manager.cc:1939] T 59536dfd577c4c199bc92891160642f9 P 6bdc477b990644489fce0187cd7b70ea: Deleting consensus metadata
I20260619 14:42:49.139209 2736 ts_tablet_manager.cc:1939] T 59536dfd577c4c199bc92891160642f9 P 00177649bdaf493a8a03c6ebba1d2d09: Deleting consensus metadata
I20260619 14:42:49.139815 2734 ts_tablet_manager.cc:1905] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:42:49.139979 2127 catalog_manager.cc:4841] TS 6bdc477b990644489fce0187cd7b70ea (127.0.98.3:32995): tablet 59536dfd577c4c199bc92891160642f9 (table test-table [id=b96dd1106870497581c0735943b4a6c4]) successfully deleted
I20260619 14:42:49.140327 2128 catalog_manager.cc:4841] TS 00177649bdaf493a8a03c6ebba1d2d09 (127.0.98.2:45007): tablet 59536dfd577c4c199bc92891160642f9 (table test-table [id=b96dd1106870497581c0735943b4a6c4]) successfully deleted
I20260619 14:42:49.142937 2734 ts_tablet_manager.cc:1918] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:42:49.143000 2734 log.cc:1196] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/wal/wals/59536dfd577c4c199bc92891160642f9
I20260619 14:42:49.143267 2734 ts_tablet_manager.cc:1939] T 59536dfd577c4c199bc92891160642f9 P 28125b7821d2490897d0b36297a2b239: Deleting consensus metadata
I20260619 14:42:49.143936 2125 catalog_manager.cc:4841] TS 28125b7821d2490897d0b36297a2b239 (127.0.98.1:45369): tablet 59536dfd577c4c199bc92891160642f9 (table test-table [id=b96dd1106870497581c0735943b4a6c4]) successfully deleted
Jun 19 14:42:49 dist-test-slave-cr22 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880169, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:42:49 dist-test-slave-cr22 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880169, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:49.170346 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:49.155489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33782 (local address 127.0.98.62:35575)
0619 14:42:49.155881 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:49.155887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:49.155901 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:42:49.155942 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:49.155944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:49.155985 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:49.156063 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:49.156900 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:49.157598 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:49.158628 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:49.158860 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:49.158944 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:49.167003 (+ 8059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:49.167006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:49.167008 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:49.167037 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:49.168645 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:49.169051 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:49.169054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:49.169055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:49.169130 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:49.169423 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:49.169426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:49.169427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:49.169611 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:49.169804 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:49.170044 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 14:42:49.170144 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":128,"threads_started":1}
I20260619 14:42:49.173126 2135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:42:49.173607 2135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:42:49.178619 2278 tablet_service.cc:1467] Processing CreateTablet for tablet d63fbc2e08094da5a25d736c77e6cd53 (DEFAULT_TABLE table=test-table [id=22cad9f904e7451f91a373d9acecd1ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:49.178632 2418 tablet_service.cc:1467] Processing CreateTablet for tablet d63fbc2e08094da5a25d736c77e6cd53 (DEFAULT_TABLE table=test-table [id=22cad9f904e7451f91a373d9acecd1ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:49.178726 2558 tablet_service.cc:1467] Processing CreateTablet for tablet d63fbc2e08094da5a25d736c77e6cd53 (DEFAULT_TABLE table=test-table [id=22cad9f904e7451f91a373d9acecd1ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:49.178929 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63fbc2e08094da5a25d736c77e6cd53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:49.178928 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63fbc2e08094da5a25d736c77e6cd53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:49.178961 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63fbc2e08094da5a25d736c77e6cd53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:49.181061 2745 tablet_bootstrap.cc:492] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239: Bootstrap starting.
I20260619 14:42:49.181972 2747 tablet_bootstrap.cc:492] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea: Bootstrap starting.
I20260619 14:42:49.182031 2745 tablet_bootstrap.cc:654] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:49.182763 2747 tablet_bootstrap.cc:654] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:49.183207 2745 tablet_bootstrap.cc:492] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239: No bootstrap required, opened a new log
I20260619 14:42:49.183298 2745 ts_tablet_manager.cc:1397] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:42:49.183336 2746 tablet_bootstrap.cc:492] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09: Bootstrap starting.
I20260619 14:42:49.183817 2745 raft_consensus.cc:348] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:49.183946 2745 raft_consensus.cc:374] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:49.183990 2745 raft_consensus.cc:729] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28125b7821d2490897d0b36297a2b239, State: Initialized, Role: FOLLOWER
I20260619 14:42:49.183902 2747 tablet_bootstrap.cc:492] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea: No bootstrap required, opened a new log
I20260619 14:42:49.184182 2747 ts_tablet_manager.cc:1397] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:42:49.184139 2745 consensus_queue.cc:260] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:49.184675 2746 tablet_bootstrap.cc:654] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:49.184655 2747 raft_consensus.cc:348] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:49.184872 2747 raft_consensus.cc:374] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:49.184947 2747 raft_consensus.cc:729] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bdc477b990644489fce0187cd7b70ea, State: Initialized, Role: FOLLOWER
I20260619 14:42:49.184962 2745 ts_tablet_manager.cc:1428] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:42:49.185139 2747 consensus_queue.cc:260] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:49.185456 2747 ts_tablet_manager.cc:1428] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:42:49.186344 2746 tablet_bootstrap.cc:492] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09: No bootstrap required, opened a new log
I20260619 14:42:49.186430 2746 ts_tablet_manager.cc:1397] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:42:49.186898 2746 raft_consensus.cc:348] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:49.187023 2746 raft_consensus.cc:374] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:49.187059 2746 raft_consensus.cc:729] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00177649bdaf493a8a03c6ebba1d2d09, State: Initialized, Role: FOLLOWER
I20260619 14:42:49.187202 2746 consensus_queue.cc:260] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:49.187615 2746 ts_tablet_manager.cc:1428] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 14:42:49.447180 2729 raft_consensus.cc:482] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:42:49.447359 2729 raft_consensus.cc:504] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:49.447947 2729 leader_election.cc:288] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00177649bdaf493a8a03c6ebba1d2d09 (127.0.98.2:45007), 6bdc477b990644489fce0187cd7b70ea (127.0.98.3:32995)
I20260619 14:42:49.448374 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63fbc2e08094da5a25d736c77e6cd53" candidate_uuid: "28125b7821d2490897d0b36297a2b239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bdc477b990644489fce0187cd7b70ea" is_pre_election: true
I20260619 14:42:49.448535 2578 raft_consensus.cc:2454] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28125b7821d2490897d0b36297a2b239 in term 0.
I20260619 14:42:49.448623 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63fbc2e08094da5a25d736c77e6cd53" candidate_uuid: "28125b7821d2490897d0b36297a2b239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00177649bdaf493a8a03c6ebba1d2d09" is_pre_election: true
I20260619 14:42:49.448799 2438 raft_consensus.cc:2454] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28125b7821d2490897d0b36297a2b239 in term 0.
I20260619 14:42:49.448879 2231 leader_election.cc:302] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28125b7821d2490897d0b36297a2b239, 6bdc477b990644489fce0187cd7b70ea; no voters:
I20260619 14:42:49.449069 2729 raft_consensus.cc:2781] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:42:49.449128 2729 raft_consensus.cc:482] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:42:49.449179 2729 raft_consensus.cc:3037] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:49.450433 2729 raft_consensus.cc:504] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:49.450762 2729 leader_election.cc:288] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [CANDIDATE]: Term 1 election: Requested vote from peers 00177649bdaf493a8a03c6ebba1d2d09 (127.0.98.2:45007), 6bdc477b990644489fce0187cd7b70ea (127.0.98.3:32995)
I20260619 14:42:49.451220 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63fbc2e08094da5a25d736c77e6cd53" candidate_uuid: "28125b7821d2490897d0b36297a2b239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00177649bdaf493a8a03c6ebba1d2d09"
I20260619 14:42:49.451224 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63fbc2e08094da5a25d736c77e6cd53" candidate_uuid: "28125b7821d2490897d0b36297a2b239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bdc477b990644489fce0187cd7b70ea"
I20260619 14:42:49.451347 2438 raft_consensus.cc:3037] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:49.451385 2578 raft_consensus.cc:3037] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:49.452298 2578 raft_consensus.cc:2454] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28125b7821d2490897d0b36297a2b239 in term 1.
I20260619 14:42:49.452515 2438 raft_consensus.cc:2454] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28125b7821d2490897d0b36297a2b239 in term 1.
I20260619 14:42:49.452598 2231 leader_election.cc:302] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28125b7821d2490897d0b36297a2b239, 6bdc477b990644489fce0187cd7b70ea; no voters:
I20260619 14:42:49.452775 2729 raft_consensus.cc:2781] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:42:49.452862 2729 raft_consensus.cc:686] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [term 1 LEADER]: Becoming Leader. State: Replica: 28125b7821d2490897d0b36297a2b239, State: Running, Role: LEADER
I20260619 14:42:49.452997 2729 consensus_queue.cc:237] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:49.454871 2141 catalog_manager.cc:5495] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28125b7821d2490897d0b36297a2b239 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "28125b7821d2490897d0b36297a2b239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } health_report { overall_health: UNKNOWN } } }
I20260619 14:42:49.488356 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:49.484269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47544 (local address 127.0.98.1:45369)
0619 14:42:49.484599 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:49.484606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:49.484619 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:42:49.484659 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:49.484661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:49.484790 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:49.484875 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:42:49.485196 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:49.485832 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:49.486945 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:49.487166 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:49.487249 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:49.487532 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:42:49.487617 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:42:49.487676 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:49.488044 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 14:42:49.488146 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":149,"threads_started":1}
I20260619 14:42:49.495719 2438 raft_consensus.cc:1261] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09 [term 1 FOLLOWER]: Refusing update from remote peer 28125b7821d2490897d0b36297a2b239: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:42:49.495751 2578 raft_consensus.cc:1261] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea [term 1 FOLLOWER]: Refusing update from remote peer 28125b7821d2490897d0b36297a2b239: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:42:49.496106 2679 consensus_queue.cc:1035] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:42:49.496258 2729 consensus_queue.cc:1035] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:42:49.530061 2141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:42:49.530262 2141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:42:49.533030 2141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb: Sending DeleteTablet for 3 replicas of tablet d63fbc2e08094da5a25d736c77e6cd53
I20260619 14:42:49.533644 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet d63fbc2e08094da5a25d736c77e6cd53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:42:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47526
I20260619 14:42:49.533949 2734 tablet_replica.cc:330] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239: stopping tablet replica
I20260619 14:42:49.534081 2734 raft_consensus.cc:2229] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:42:49.534037 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet d63fbc2e08094da5a25d736c77e6cd53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:42:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60070
I20260619 14:42:49.534292 2735 tablet_replica.cc:330] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea: stopping tablet replica
I20260619 14:42:49.534358 2734 raft_consensus.cc:2258] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:42:49.534440 2735 raft_consensus.cc:2229] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:42:49.534574 2735 raft_consensus.cc:2258] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:42:49.534557 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet d63fbc2e08094da5a25d736c77e6cd53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:42:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50680
I20260619 14:42:49.534759 2736 tablet_replica.cc:330] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09: stopping tablet replica
I20260619 14:42:49.534883 2736 raft_consensus.cc:2229] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:42:49.534822 2141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 14:42:49.535033 2736 raft_consensus.cc:2258] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 14:42:49.535251 2141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:42:49.535384 2734 ts_tablet_manager.cc:1905] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:42:49.535810 2735 ts_tablet_manager.cc:1905] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:42:49.535818 2736 ts_tablet_manager.cc:1905] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:42:49.538259 2736 ts_tablet_manager.cc:1918] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:42:49.538259 2735 ts_tablet_manager.cc:1918] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:42:49.538342 2736 log.cc:1196] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/d63fbc2e08094da5a25d736c77e6cd53
I20260619 14:42:49.538342 2735 log.cc:1196] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/d63fbc2e08094da5a25d736c77e6cd53
I20260619 14:42:49.538645 2736 ts_tablet_manager.cc:1939] T d63fbc2e08094da5a25d736c77e6cd53 P 00177649bdaf493a8a03c6ebba1d2d09: Deleting consensus metadata
I20260619 14:42:49.538703 2735 ts_tablet_manager.cc:1939] T d63fbc2e08094da5a25d736c77e6cd53 P 6bdc477b990644489fce0187cd7b70ea: Deleting consensus metadata
I20260619 14:42:49.539157 2734 ts_tablet_manager.cc:1918] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:42:49.539332 2734 log.cc:1196] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-0/wal/wals/d63fbc2e08094da5a25d736c77e6cd53
I20260619 14:42:49.539502 2128 catalog_manager.cc:4841] TS 00177649bdaf493a8a03c6ebba1d2d09 (127.0.98.2:45007): tablet d63fbc2e08094da5a25d736c77e6cd53 (table test-table [id=22cad9f904e7451f91a373d9acecd1ca]) successfully deleted
I20260619 14:42:49.539935 2734 ts_tablet_manager.cc:1939] T d63fbc2e08094da5a25d736c77e6cd53 P 28125b7821d2490897d0b36297a2b239: Deleting consensus metadata
I20260619 14:42:49.540079 2127 catalog_manager.cc:4841] TS 6bdc477b990644489fce0187cd7b70ea (127.0.98.3:32995): tablet d63fbc2e08094da5a25d736c77e6cd53 (table test-table [id=22cad9f904e7451f91a373d9acecd1ca]) successfully deleted
I20260619 14:42:49.541075 2125 catalog_manager.cc:4841] TS 28125b7821d2490897d0b36297a2b239 (127.0.98.1:45369): tablet d63fbc2e08094da5a25d736c77e6cd53 (table test-table [id=22cad9f904e7451f91a373d9acecd1ca]) successfully deleted
I20260619 14:42:49.541292 2418 tablet_service.cc:1467] Processing CreateTablet for tablet ea8f7c6c183d45c8b7bfacedcc33df90 (DEFAULT_TABLE table=test-table [id=b882d9e9022a4a15914707e508a8a2a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:49.541575 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea8f7c6c183d45c8b7bfacedcc33df90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:49.541553 2558 tablet_service.cc:1467] Processing CreateTablet for tablet ea8f7c6c183d45c8b7bfacedcc33df90 (DEFAULT_TABLE table=test-table [id=b882d9e9022a4a15914707e508a8a2a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:49.541711 2278 tablet_service.cc:1467] Processing CreateTablet for tablet ea8f7c6c183d45c8b7bfacedcc33df90 (DEFAULT_TABLE table=test-table [id=b882d9e9022a4a15914707e508a8a2a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:42:49.541802 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea8f7c6c183d45c8b7bfacedcc33df90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:49.541945 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea8f7c6c183d45c8b7bfacedcc33df90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:49.543993 2747 tablet_bootstrap.cc:492] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea: Bootstrap starting.
I20260619 14:42:49.544291 2745 tablet_bootstrap.cc:492] T ea8f7c6c183d45c8b7bfacedcc33df90 P 28125b7821d2490897d0b36297a2b239: Bootstrap starting.
I20260619 14:42:49.544775 2747 tablet_bootstrap.cc:654] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:49.545004 2745 tablet_bootstrap.cc:654] T ea8f7c6c183d45c8b7bfacedcc33df90 P 28125b7821d2490897d0b36297a2b239: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:49.545325 2746 tablet_bootstrap.cc:492] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09: Bootstrap starting.
I20260619 14:42:49.545831 2747 tablet_bootstrap.cc:492] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea: No bootstrap required, opened a new log
I20260619 14:42:49.545921 2747 ts_tablet_manager.cc:1397] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:42:49.546133 2745 tablet_bootstrap.cc:492] T ea8f7c6c183d45c8b7bfacedcc33df90 P 28125b7821d2490897d0b36297a2b239: No bootstrap required, opened a new log
I20260619 14:42:49.546241 2745 ts_tablet_manager.cc:1397] T ea8f7c6c183d45c8b7bfacedcc33df90 P 28125b7821d2490897d0b36297a2b239: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:42:49.546334 2746 tablet_bootstrap.cc:654] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:49.546396 2747 raft_consensus.cc:348] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:49.546541 2747 raft_consensus.cc:374] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:49.546573 2747 raft_consensus.cc:729] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bdc477b990644489fce0187cd7b70ea, State: Initialized, Role: FOLLOWER
I20260619 14:42:49.546653 2745 raft_consensus.cc:348] T ea8f7c6c183d45c8b7bfacedcc33df90 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:49.546706 2747 consensus_queue.cc:260] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:49.546756 2745 raft_consensus.cc:374] T ea8f7c6c183d45c8b7bfacedcc33df90 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:49.546797 2745 raft_consensus.cc:729] T ea8f7c6c183d45c8b7bfacedcc33df90 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28125b7821d2490897d0b36297a2b239, State: Initialized, Role: FOLLOWER
I20260619 14:42:49.546923 2745 consensus_queue.cc:260] T ea8f7c6c183d45c8b7bfacedcc33df90 P 28125b7821d2490897d0b36297a2b239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:49.547047 2747 ts_tablet_manager.cc:1428] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:42:49.547209 2745 ts_tablet_manager.cc:1428] T ea8f7c6c183d45c8b7bfacedcc33df90 P 28125b7821d2490897d0b36297a2b239: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:42:49.547796 2746 tablet_bootstrap.cc:492] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09: No bootstrap required, opened a new log
I20260619 14:42:49.547904 2746 ts_tablet_manager.cc:1397] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:42:49.548355 2746 raft_consensus.cc:348] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:49.548476 2746 raft_consensus.cc:374] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:49.548516 2746 raft_consensus.cc:729] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00177649bdaf493a8a03c6ebba1d2d09, State: Initialized, Role: FOLLOWER
I20260619 14:42:49.548652 2746 consensus_queue.cc:260] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:49.549052 2746 ts_tablet_manager.cc:1428] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:42:49.855852 2715 raft_consensus.cc:482] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:42:49.856002 2715 raft_consensus.cc:504] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:49.856467 2715 leader_election.cc:288] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28125b7821d2490897d0b36297a2b239 (127.0.98.1:45369), 00177649bdaf493a8a03c6ebba1d2d09 (127.0.98.2:45007)
I20260619 14:42:49.857013 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8f7c6c183d45c8b7bfacedcc33df90" candidate_uuid: "6bdc477b990644489fce0187cd7b70ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00177649bdaf493a8a03c6ebba1d2d09" is_pre_election: true
I20260619 14:42:49.857069 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8f7c6c183d45c8b7bfacedcc33df90" candidate_uuid: "6bdc477b990644489fce0187cd7b70ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28125b7821d2490897d0b36297a2b239" is_pre_election: true
I20260619 14:42:49.857182 2438 raft_consensus.cc:2454] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bdc477b990644489fce0187cd7b70ea in term 0.
I20260619 14:42:49.857235 2298 raft_consensus.cc:2454] T ea8f7c6c183d45c8b7bfacedcc33df90 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bdc477b990644489fce0187cd7b70ea in term 0.
I20260619 14:42:49.857509 2512 leader_election.cc:302] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00177649bdaf493a8a03c6ebba1d2d09, 6bdc477b990644489fce0187cd7b70ea; no voters:
I20260619 14:42:49.857702 2715 raft_consensus.cc:2781] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:42:49.857774 2715 raft_consensus.cc:482] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:42:49.857818 2715 raft_consensus.cc:3037] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:49.858772 2715 raft_consensus.cc:504] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:49.859083 2715 leader_election.cc:288] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [CANDIDATE]: Term 1 election: Requested vote from peers 28125b7821d2490897d0b36297a2b239 (127.0.98.1:45369), 00177649bdaf493a8a03c6ebba1d2d09 (127.0.98.2:45007)
I20260619 14:42:49.859478 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8f7c6c183d45c8b7bfacedcc33df90" candidate_uuid: "6bdc477b990644489fce0187cd7b70ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00177649bdaf493a8a03c6ebba1d2d09"
I20260619 14:42:49.859539 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8f7c6c183d45c8b7bfacedcc33df90" candidate_uuid: "6bdc477b990644489fce0187cd7b70ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28125b7821d2490897d0b36297a2b239"
I20260619 14:42:49.859607 2438 raft_consensus.cc:3037] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:49.859678 2298 raft_consensus.cc:3037] T ea8f7c6c183d45c8b7bfacedcc33df90 P 28125b7821d2490897d0b36297a2b239 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:49.860592 2298 raft_consensus.cc:2454] T ea8f7c6c183d45c8b7bfacedcc33df90 P 28125b7821d2490897d0b36297a2b239 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bdc477b990644489fce0187cd7b70ea in term 1.
I20260619 14:42:49.860605 2438 raft_consensus.cc:2454] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bdc477b990644489fce0187cd7b70ea in term 1.
I20260619 14:42:49.860888 2509 leader_election.cc:302] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28125b7821d2490897d0b36297a2b239, 6bdc477b990644489fce0187cd7b70ea; no voters:
I20260619 14:42:49.861136 2715 raft_consensus.cc:2781] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:42:49.861212 2715 raft_consensus.cc:686] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [term 1 LEADER]: Becoming Leader. State: Replica: 6bdc477b990644489fce0187cd7b70ea, State: Running, Role: LEADER
I20260619 14:42:49.861342 2715 consensus_queue.cc:237] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } }
I20260619 14:42:49.863058 2141 catalog_manager.cc:5495] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bdc477b990644489fce0187cd7b70ea (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6bdc477b990644489fce0187cd7b70ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bdc477b990644489fce0187cd7b70ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32995 } health_report { overall_health: HEALTHY } } }
I20260619 14:42:49.921586 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:49.918066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60084 (local address 127.0.98.3:32995)
0619 14:42:49.918384 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:49.918401 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:49.918413 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:42:49.918452 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:49.918454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:49.918495 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:49.918594 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:42:49.918865 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:49.919474 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:49.920412 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:49.920668 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:49.920781 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:49.921060 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:42:49.921136 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:42:49.921224 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:49.921352 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 14:42:49.921408 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":119,"threads_started":1}
I20260619 14:42:49.931596 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:49.927849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34079 (local address 127.0.98.3:32995)
0619 14:42:49.927965 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:49.927969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:49.928070 (+ 101us) server_negotiation.cc:407] Connection header received
0619 14:42:49.928189 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:49.928196 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:49.928235 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:49.928300 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:42:49.928596 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:49.929200 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:49.930492 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:49.930756 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:49.930894 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:49.931130 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:42:49.931201 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:42:49.931282 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:49.931362 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 14:42:49.931409 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":44}
I20260619 14:42:49.931596 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:49.927675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32995 (local address 127.0.98.3:34079)
0619 14:42:49.927984 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:49.927999 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:49.928099 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:49.928337 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:49.928340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:49.928348 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:42:49.928463 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:49.928469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:49.929311 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:49.929314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:49.930364 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:49.930374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:49.930893 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:49.930897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:49.930988 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:49.931018 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:42:49.931284 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:42:49.931292 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:42:49.931355 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 14:42:49.931428 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
I20260619 14:42:49.937263 2298 raft_consensus.cc:1261] T ea8f7c6c183d45c8b7bfacedcc33df90 P 28125b7821d2490897d0b36297a2b239 [term 1 FOLLOWER]: Refusing update from remote peer 6bdc477b990644489fce0187cd7b70ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:42:49.937290 2437 raft_consensus.cc:1261] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09 [term 1 FOLLOWER]: Refusing update from remote peer 6bdc477b990644489fce0187cd7b70ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:42:49.937911 2715 consensus_queue.cc:1035] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "00177649bdaf493a8a03c6ebba1d2d09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:42:49.938090 2686 consensus_queue.cc:1035] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "28125b7821d2490897d0b36297a2b239" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:42:49.953553 2713 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 14:42:51.506577 2135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:42:51.506742 2135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:42:51.509305 2135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee86775e38c646c5a6796977f2f537bb: Sending DeleteTablet for 3 replicas of tablet ea8f7c6c183d45c8b7bfacedcc33df90
I20260619 14:42:51.509940 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet ea8f7c6c183d45c8b7bfacedcc33df90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:42:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47526
I20260619 14:42:51.510133 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet ea8f7c6c183d45c8b7bfacedcc33df90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:42:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50680
I20260619 14:42:51.510322 2789 tablet_replica.cc:330] T ea8f7c6c183d45c8b7bfacedcc33df90 P 28125b7821d2490897d0b36297a2b239: stopping tablet replica
I20260619 14:42:51.510347 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet ea8f7c6c183d45c8b7bfacedcc33df90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:42:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60070
I20260619 14:42:51.510437 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 2213
I20260619 14:42:51.511248 2790 tablet_replica.cc:330] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09: stopping tablet replica
I20260619 14:42:51.511240 2791 tablet_replica.cc:330] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea: stopping tablet replica
I20260619 14:42:51.511356 2791 raft_consensus.cc:2229] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:42:51.511406 2790 raft_consensus.cc:2229] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:42:51.511579 2790 raft_consensus.cc:2258] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:42:51.512137 2791 raft_consensus.cc:2258] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:42:51.512428 2790 ts_tablet_manager.cc:1905] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:42:51.512933 2791 ts_tablet_manager.cc:1905] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:42:51.515192 2790 ts_tablet_manager.cc:1918] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:42:51.515280 2790 log.cc:1196] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/ea8f7c6c183d45c8b7bfacedcc33df90
I20260619 14:42:51.515599 2790 ts_tablet_manager.cc:1939] T ea8f7c6c183d45c8b7bfacedcc33df90 P 00177649bdaf493a8a03c6ebba1d2d09: Deleting consensus metadata
I20260619 14:42:51.515661 2791 ts_tablet_manager.cc:1918] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:42:51.515738 2791 log.cc:1196] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/ea8f7c6c183d45c8b7bfacedcc33df90
I20260619 14:42:51.516022 2791 ts_tablet_manager.cc:1939] T ea8f7c6c183d45c8b7bfacedcc33df90 P 6bdc477b990644489fce0187cd7b70ea: Deleting consensus metadata
I20260619 14:42:51.516407 2128 catalog_manager.cc:4841] TS 00177649bdaf493a8a03c6ebba1d2d09 (127.0.98.2:45007): tablet ea8f7c6c183d45c8b7bfacedcc33df90 (table test-table [id=b882d9e9022a4a15914707e508a8a2a5]) successfully deleted
I20260619 14:42:51.516739 2127 catalog_manager.cc:4841] TS 6bdc477b990644489fce0187cd7b70ea (127.0.98.3:32995): tablet ea8f7c6c183d45c8b7bfacedcc33df90 (table test-table [id=b882d9e9022a4a15914707e508a8a2a5]) successfully deleted
W20260619 14:42:51.518752 2125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45369 (error 108)
I20260619 14:42:51.519868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 2353
I20260619 14:42:51.520349 2793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:51.519639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45369 (local address 127.0.0.1:47554)
0619 14:42:51.520025 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:51.520167 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":84,"threads_started":1}
W20260619 14:42:51.520694 2125 catalog_manager.cc:4584] TS 28125b7821d2490897d0b36297a2b239 (127.0.98.1:45369): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea8f7c6c183d45c8b7bfacedcc33df90: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45369: connect: Connection refused (error 111)
I20260619 14:42:51.526770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 2493
I20260619 14:42:51.533974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 2109
2026-06-19T14:42:51Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7060 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:42:51 dist-test-slave-cr22 krb5kdc[2796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:42:51 dist-test-slave-cr22 krb5kdc[2796](info): set up 2 sockets
Jun 19 14:42:51 dist-test-slave-cr22 krb5kdc[2796](info): commencing operation
krb5kdc: starting...
W20260619 14:42:52.603507 392 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:42:52 dist-test-slave-cr22 krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:42:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:42:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:42:52.730831 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42533
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:52.841485 2809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:52.841898 2809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:52.841981 2809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:52.846169 2809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:42:52.846289 2809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:52.846345 2809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:52.846411 2809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:42:52.846448 2809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:42:52.851754 2809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:52.852896 2809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:52.853956 2809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:52.862637 2814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:52.862666 2817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:52.862666 2815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:52.862936 2809 server_base.cc:1034] running on GCE node
I20260619 14:42:52.863621 2809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:52.864648 2809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:52.865846 2809 hybrid_clock.cc:648] HybridClock initialized: now 1781880172865814 us; error 52 us; skew 500 ppm
Jun 19 14:42:52 dist-test-slave-cr22 krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:52.869047 2809 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:42:52.869870 2809 webserver.cc:460] Webserver started at http://127.0.98.62:40139/ using document root <none> and password file <none>
I20260619 14:42:52.870409 2809 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:52.870496 2809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:52.870725 2809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:52.872743 2809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "6c0598901cb34bb6ae1d3e9ea9fc2574"
format_stamp: "Formatted at 2026-06-19 14:42:52 on dist-test-slave-cr22"
server_key: "f646fbbb7b1d28dde6e3b6dcfd7c27a8"
server_key_iv: "6107cf09266c447ba3e9c157b95533f6"
server_key_version: "encryptionkey@0"
I20260619 14:42:52.873287 2809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "6c0598901cb34bb6ae1d3e9ea9fc2574"
format_stamp: "Formatted at 2026-06-19 14:42:52 on dist-test-slave-cr22"
server_key: "f646fbbb7b1d28dde6e3b6dcfd7c27a8"
server_key_iv: "6107cf09266c447ba3e9c157b95533f6"
server_key_version: "encryptionkey@0"
I20260619 14:42:52.877022 2809 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:42:52.879323 2824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:52.880546 2809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:42:52.880684 2809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "6c0598901cb34bb6ae1d3e9ea9fc2574"
format_stamp: "Formatted at 2026-06-19 14:42:52 on dist-test-slave-cr22"
server_key: "f646fbbb7b1d28dde6e3b6dcfd7c27a8"
server_key_iv: "6107cf09266c447ba3e9c157b95533f6"
server_key_version: "encryptionkey@0"
I20260619 14:42:52.880790 2809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:52.919720 2809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:52.922618 2809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:52.922825 2809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:52.929953 2809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42533
I20260619 14:42:52.929970 2876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42533 every 8 connection(s)
I20260619 14:42:52.930881 2809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:42:52.933605 2877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:52.937882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 2809
I20260619 14:42:52.937994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:42:52.938265 392 external_mini_cluster.cc:1428] Setting key dc6cd191513702f7ccc99cf6d7560d82
I20260619 14:42:52.939623 2877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574: Bootstrap starting.
I20260619 14:42:52.942022 2877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:52.942809 2877 log.cc:824] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574: Log is configured to *not* fsync() on all Append() calls
I20260619 14:42:52.944687 2877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574: No bootstrap required, opened a new log
Jun 19 14:42:52 dist-test-slave-cr22 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:52.949122 2877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0598901cb34bb6ae1d3e9ea9fc2574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42533 } }
I20260619 14:42:52.949435 2877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:52.949548 2877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0598901cb34bb6ae1d3e9ea9fc2574, State: Initialized, Role: FOLLOWER
I20260619 14:42:52.950119 2877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0598901cb34bb6ae1d3e9ea9fc2574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42533 } }
I20260619 14:42:52.950323 2877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:42:52.950409 2877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:42:52.950524 2877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:52.951850 2877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0598901cb34bb6ae1d3e9ea9fc2574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42533 } }
I20260619 14:42:52.952214 2877 leader_election.cc:302] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c0598901cb34bb6ae1d3e9ea9fc2574; no voters:
I20260619 14:42:52.952560 2877 leader_election.cc:288] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:42:52.953823 2877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:42:52.953940 2880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:52.939660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36450 (local address 127.0.98.62:42533)
0619 14:42:52.940111 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:52.940126 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:52.940160 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:42:52.940799 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:52.940815 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:52.941129 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:52.941412 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:52.942344 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:52.943413 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:52.944355 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:52.944916 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:52.945097 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:52.948146 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:52.948153 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:52.948172 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:52.948214 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:52.950439 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:52.951001 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:52.951006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:52.951011 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:52.951137 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:52.951567 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:52.951570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:52.951572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:52.952109 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:52.952317 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:52.952702 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 14:42:52.953007 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":147,"threads_started":1}
I20260619 14:42:52.953967 2883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:42:52.954154 2883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574 [term 1 LEADER]: Becoming Leader. State: Replica: 6c0598901cb34bb6ae1d3e9ea9fc2574, State: Running, Role: LEADER
I20260619 14:42:52.954434 2883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0598901cb34bb6ae1d3e9ea9fc2574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42533 } }
I20260619 14:42:52.955698 2882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c0598901cb34bb6ae1d3e9ea9fc2574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0598901cb34bb6ae1d3e9ea9fc2574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42533 } } }
I20260619 14:42:52.955813 2882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574 [sys.catalog]: This master's current role is: LEADER
I20260619 14:42:52.955739 2883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c0598901cb34bb6ae1d3e9ea9fc2574. Latest consensus state: current_term: 1 leader_uuid: "6c0598901cb34bb6ae1d3e9ea9fc2574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0598901cb34bb6ae1d3e9ea9fc2574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42533 } } }
I20260619 14:42:52.955936 2883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574 [sys.catalog]: This master's current role is: LEADER
W20260619 14:42:52.959493 2896 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:42:52.959565 2896 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:42:52.959617 2897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:42:52.960429 2897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:42:52.966214 2897 catalog_manager.cc:1347] Generated new cluster ID: c52cceb3096f48c88a085cf5e20f9f5d
I20260619 14:42:52.966285 2897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:42:53.001175 2897 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:42:53.002626 2897 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:42:53.012406 2897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c0598901cb34bb6ae1d3e9ea9fc2574: Generated new TSK 0
I20260619 14:42:53.013140 2897 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260619 14:42:53.324599 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.298s user 0.000s sys 0.004s
I20260619 14:42:53.325115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42533
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:53.435050 2903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:53.435297 2903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:53.435339 2903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:53.439368 2903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:53.439460 2903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:53.439522 2903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:42:53.444746 2903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:53.445765 2903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:53.446717 2903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:53.456789 2911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:53.456784 2909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:53.456789 2908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:53.457082 2903 server_base.cc:1034] running on GCE node
I20260619 14:42:53.457522 2903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:53.458204 2903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:53.459420 2903 hybrid_clock.cc:648] HybridClock initialized: now 1781880173459399 us; error 36 us; skew 500 ppm
Jun 19 14:42:53 dist-test-slave-cr22 krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:53.462437 2903 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:42:53.463277 2903 webserver.cc:460] Webserver started at http://127.0.98.1:41707/ using document root <none> and password file <none>
I20260619 14:42:53.463904 2903 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:53.463996 2903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:53.464208 2903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:53.466276 2903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "01b249ec5f2c4d478b7e1af413ef9f98"
format_stamp: "Formatted at 2026-06-19 14:42:53 on dist-test-slave-cr22"
server_key: "1e1ba299815053b7a7e2533a86725771"
server_key_iv: "aaea531d3c0afaa29307711fe26ba96f"
server_key_version: "encryptionkey@0"
I20260619 14:42:53.466850 2903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "01b249ec5f2c4d478b7e1af413ef9f98"
format_stamp: "Formatted at 2026-06-19 14:42:53 on dist-test-slave-cr22"
server_key: "1e1ba299815053b7a7e2533a86725771"
server_key_iv: "aaea531d3c0afaa29307711fe26ba96f"
server_key_version: "encryptionkey@0"
I20260619 14:42:53.470744 2903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 14:42:53.473301 2918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:53.474553 2903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:42:53.474710 2903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "01b249ec5f2c4d478b7e1af413ef9f98"
format_stamp: "Formatted at 2026-06-19 14:42:53 on dist-test-slave-cr22"
server_key: "1e1ba299815053b7a7e2533a86725771"
server_key_iv: "aaea531d3c0afaa29307711fe26ba96f"
server_key_version: "encryptionkey@0"
I20260619 14:42:53.474834 2903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:53.498164 2903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:53.501643 2903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:53.501886 2903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:53.502501 2903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:42:53.503580 2903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:53.503705 2903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:53.503769 2903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:53.503796 2903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:53.513154 2903 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46697
I20260619 14:42:53.513173 3031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46697 every 8 connection(s)
I20260619 14:42:53.514046 2903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:42:53.521700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 2903
I20260619 14:42:53.521879 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:42:53.522181 392 external_mini_cluster.cc:1428] Setting key 343188b3ab7a799d8dc87910ac587d5b
Jun 19 14:42:53 dist-test-slave-cr22 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:53.529660 3034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:53.516088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44605 (local address 127.0.98.62:42533)
0619 14:42:53.516386 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:53.516389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:53.517022 (+ 633us) server_negotiation.cc:407] Connection header received
0619 14:42:53.518433 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:53.518436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:53.518493 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:53.518611 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:53.519830 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:53.520409 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:53.521539 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:53.521817 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:53.521948 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:53.525048 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:53.525055 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:53.525061 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:53.525095 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:53.526952 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:53.527660 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:53.527663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:53.527664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:53.527749 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:53.528376 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:53.528379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:53.528380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:53.528569 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:53.528651 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:53.529405 (+ 754us) server_negotiation.cc:299] Negotiation successful
0619 14:42:53.529523 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":116,"threads_started":1}
I20260619 14:42:53.530867 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:53.516278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42533 (local address 127.0.98.1:44605)
0619 14:42:53.516720 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:53.516896 (+ 176us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:53.518215 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:53.518778 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:53.518787 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:53.519166 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:53.519667 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:53.519679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:53.520538 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:53.520541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:53.521410 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:53.521419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:53.522011 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:53.522015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:53.522176 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:53.522978 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:53.523017 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:53.524873 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:53.527096 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:53.527106 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:53.527126 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:53.527554 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:53.527992 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:53.527998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:53.528003 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:53.528216 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:53.528687 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:53.528694 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:53.529070 (+ 376us) client_negotiation.cc:769] Sending connection context
0619 14:42:53.529321 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 14:42:53.529728 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:42:53.532670 3032 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42533
I20260619 14:42:53.533015 3032 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:53.533550 3032 heartbeater.cc:511] Master 127.0.98.62:42533 requested a full tablet report, sending...
I20260619 14:42:53.535780 2841 ts_manager.cc:194] Registered new tserver with Master: 01b249ec5f2c4d478b7e1af413ef9f98 (127.0.98.1:46697)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:42:53.537755 2841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:42:53.556236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42533
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:53.668282 3038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:53.668572 3038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:53.668617 3038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:53.672907 3038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:53.672971 3038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:53.673031 3038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:42:53.678411 3038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:53.679543 3038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:53.680521 3038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:53.690482 3043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:53.690505 3044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:53.690680 3038 server_base.cc:1034] running on GCE node
W20260619 14:42:53.690953 3046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:53.691339 3038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:53.692049 3038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:53.693225 3038 hybrid_clock.cc:648] HybridClock initialized: now 1781880173693203 us; error 33 us; skew 500 ppm
Jun 19 14:42:53 dist-test-slave-cr22 krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:53.696269 3038 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:42:53.697221 3038 webserver.cc:460] Webserver started at http://127.0.98.2:37471/ using document root <none> and password file <none>
I20260619 14:42:53.697830 3038 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:53.697902 3038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:53.698158 3038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:53.700421 3038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "eb2c91a0bc1348dcaac1e1f9580ddfd6"
format_stamp: "Formatted at 2026-06-19 14:42:53 on dist-test-slave-cr22"
server_key: "0edb4e00f76c6f78567963e623bf59b4"
server_key_iv: "453af5090d94470441894e3e144e45f3"
server_key_version: "encryptionkey@0"
I20260619 14:42:53.701032 3038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eb2c91a0bc1348dcaac1e1f9580ddfd6"
format_stamp: "Formatted at 2026-06-19 14:42:53 on dist-test-slave-cr22"
server_key: "0edb4e00f76c6f78567963e623bf59b4"
server_key_iv: "453af5090d94470441894e3e144e45f3"
server_key_version: "encryptionkey@0"
I20260619 14:42:53.705158 3038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 14:42:53.708060 3053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:53.709419 3038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:42:53.709563 3038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "eb2c91a0bc1348dcaac1e1f9580ddfd6"
format_stamp: "Formatted at 2026-06-19 14:42:53 on dist-test-slave-cr22"
server_key: "0edb4e00f76c6f78567963e623bf59b4"
server_key_iv: "453af5090d94470441894e3e144e45f3"
server_key_version: "encryptionkey@0"
I20260619 14:42:53.709681 3038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:53.723662 3038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:53.727276 3038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:53.727558 3038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:53.728201 3038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:42:53.729063 3038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:53.729151 3038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:53.729252 3038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:53.729305 3038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:53.738343 3038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34957
I20260619 14:42:53.738426 3166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34957 every 8 connection(s)
I20260619 14:42:53.739291 3038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:42:53.743386 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 3038
I20260619 14:42:53.743578 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:42:53.743971 392 external_mini_cluster.cc:1428] Setting key 24f1642add4645527c5349cc0995739e
Jun 19 14:42:53 dist-test-slave-cr22 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:42:53.756851 3034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:53.741275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46607 (local address 127.0.98.62:42533)
0619 14:42:53.741513 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:53.741518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:53.742378 (+ 860us) server_negotiation.cc:407] Connection header received
0619 14:42:53.743966 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:53.743970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:53.744029 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:53.744113 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:53.745713 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:53.746311 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:53.748065 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:53.748492 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:53.748621 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:53.751805 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:53.751810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:53.751814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:53.751851 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:53.753919 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:53.754615 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:53.754619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:53.754621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:53.754715 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:53.755496 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:53.755500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:53.755502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:53.755777 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:53.755867 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:53.756521 (+ 654us) server_negotiation.cc:299] Negotiation successful
0619 14:42:53.756659 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260619 14:42:53.757884 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:53.741720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42533 (local address 127.0.98.2:46607)
0619 14:42:53.742186 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:53.742234 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:53.743752 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:53.744329 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:53.744339 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:53.744866 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:53.745502 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:53.745520 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:53.746459 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:53.746467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:53.747926 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:53.747937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:53.748684 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:53.748687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:53.748852 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:53.749613 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:53.749643 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:53.751604 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:53.754064 (+ 2460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:53.754069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:53.754084 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:53.754411 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:53.754973 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:53.754978 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:53.754980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:53.755142 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:53.755909 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:53.755915 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:53.756212 (+ 297us) client_negotiation.cc:769] Sending connection context
0619 14:42:53.756394 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 14:42:53.756753 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":127,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:42:53.759227 3167 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42533
I20260619 14:42:53.759601 3167 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:53.760419 3167 heartbeater.cc:511] Master 127.0.98.62:42533 requested a full tablet report, sending...
I20260619 14:42:53.761718 2841 ts_manager.cc:194] Registered new tserver with Master: eb2c91a0bc1348dcaac1e1f9580ddfd6 (127.0.98.2:34957)
I20260619 14:42:53.762547 2841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:42:53.780006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42533
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:53.894208 3172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:53.894490 3172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:53.894537 3172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:53.899286 3172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:53.899348 3172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:53.899449 3172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:42:53.904964 3172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:53.906023 3172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:53.907032 3172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:53.917124 3178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:53.917311 3177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:53.917289 3172 server_base.cc:1034] running on GCE node
W20260619 14:42:53.917491 3180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:53.917856 3172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:53.918542 3172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:53.919737 3172 hybrid_clock.cc:648] HybridClock initialized: now 1781880173919701 us; error 47 us; skew 500 ppm
Jun 19 14:42:53 dist-test-slave-cr22 krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:53.922502 3172 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:42:53.923511 3172 webserver.cc:460] Webserver started at http://127.0.98.3:46051/ using document root <none> and password file <none>
I20260619 14:42:53.924079 3172 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:53.924180 3172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:53.924407 3172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:53.926517 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "afcadfaecd984e2c8997b4ee8020d70d"
format_stamp: "Formatted at 2026-06-19 14:42:53 on dist-test-slave-cr22"
server_key: "0ed48ab6519ee168599864d3e8ff096f"
server_key_iv: "5e163c71de2c3389193c4c8c3a6ff0c7"
server_key_version: "encryptionkey@0"
I20260619 14:42:53.927071 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "afcadfaecd984e2c8997b4ee8020d70d"
format_stamp: "Formatted at 2026-06-19 14:42:53 on dist-test-slave-cr22"
server_key: "0ed48ab6519ee168599864d3e8ff096f"
server_key_iv: "5e163c71de2c3389193c4c8c3a6ff0c7"
server_key_version: "encryptionkey@0"
I20260619 14:42:53.930822 3172 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:42:53.933341 3187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:53.934626 3172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:42:53.934777 3172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "afcadfaecd984e2c8997b4ee8020d70d"
format_stamp: "Formatted at 2026-06-19 14:42:53 on dist-test-slave-cr22"
server_key: "0ed48ab6519ee168599864d3e8ff096f"
server_key_iv: "5e163c71de2c3389193c4c8c3a6ff0c7"
server_key_version: "encryptionkey@0"
I20260619 14:42:53.934893 3172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:53.944536 3172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:53.947310 3172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:53.947582 3172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:53.948156 3172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:42:53.948931 3172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:53.949000 3172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:53.949069 3172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:53.949126 3172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:53.958395 3172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33791
I20260619 14:42:53.958416 3300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33791 every 8 connection(s)
I20260619 14:42:53.959348 3172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:42:53.966486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 3172
I20260619 14:42:53.966583 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:42:53.966954 392 external_mini_cluster.cc:1428] Setting key 24fea09c7bb4cb4273b24ef9c2d52345
Jun 19 14:42:53 dist-test-slave-cr22 krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:53.974128 3034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:53.961313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40397 (local address 127.0.98.62:42533)
0619 14:42:53.961476 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:53.961481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:53.962276 (+ 795us) server_negotiation.cc:407] Connection header received
0619 14:42:53.963788 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:53.963790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:53.963834 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:53.963912 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:53.965080 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:53.965629 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:53.966812 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:53.967139 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:53.967266 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:53.970565 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:53.970567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:53.970569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:53.970592 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:53.972088 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:53.972600 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:53.972603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:53.972604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:53.972677 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:53.973091 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:53.973094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:53.973095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:53.973335 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:53.973424 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:53.973905 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 14:42:53.974006 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:42:53.974949 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:53.961668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42533 (local address 127.0.98.3:40397)
0619 14:42:53.962129 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:53.962170 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:53.963578 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:53.964095 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:53.964102 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:53.964486 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:53.964950 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:53.964962 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:53.965738 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:53.965741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:53.966695 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:53.966704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:53.967343 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:53.967348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:53.967584 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:53.968437 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:53.968461 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:53.970417 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:53.972238 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:53.972244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:53.972255 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:53.972505 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:53.972790 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:53.972794 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:53.972797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:53.972980 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:53.973438 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:53.973442 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:53.973662 (+ 220us) client_negotiation.cc:769] Sending connection context
0619 14:42:53.973820 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 14:42:53.974066 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":123,"threads_started":1}
I20260619 14:42:53.976219 3301 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42533
I20260619 14:42:53.976462 3301 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:53.977035 3301 heartbeater.cc:511] Master 127.0.98.62:42533 requested a full tablet report, sending...
I20260619 14:42:53.978053 2841 ts_manager.cc:194] Registered new tserver with Master: afcadfaecd984e2c8997b4ee8020d70d (127.0.98.3:33791)
I20260619 14:42:53.978645 2841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40397
I20260619 14:42:53.981261 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:42:53.995481 3034 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:42:53.993835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44044 (local address 127.0.98.62:42533)
0619 14:42:53.993975 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:53.993981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:53.994151 (+ 170us) server_negotiation.cc:407] Connection header received
0619 14:42:53.994342 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:53.994344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:53.994389 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:53.994478 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:53.995293 (+ 815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44044: BlockingRecv error: recv got EOF from 127.0.0.1:44044 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:42:53.996646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 2903
I20260619 14:42:54.002498 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 3038
I20260619 14:42:54.008369 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 3172
I20260619 14:42:54.013717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 2809
2026-06-19T14:42:54Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2477 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:42:54 dist-test-slave-cr22 krb5kdc[3313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:42:54 dist-test-slave-cr22 krb5kdc[3313](info): set up 2 sockets
Jun 19 14:42:54 dist-test-slave-cr22 krb5kdc[3313](info): commencing operation
krb5kdc: starting...
W20260619 14:42:55.063666 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:42:55 dist-test-slave-cr22 krb5kdc[3313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:42:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:42:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:42:55.190115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40659
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:55.298107 3326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:55.298435 3326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:55.298507 3326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:55.302580 3326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:42:55.302691 3326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:55.302731 3326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:55.302793 3326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:42:55.302834 3326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:42:55.308063 3326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:55.309173 3326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:55.310228 3326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:55.318933 3334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:55.319070 3331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:55.318835 3332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:55.319345 3326 server_base.cc:1034] running on GCE node
I20260619 14:42:55.320006 3326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:55.321226 3326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:55.322410 3326 hybrid_clock.cc:648] HybridClock initialized: now 1781880175322380 us; error 42 us; skew 500 ppm
Jun 19 14:42:55 dist-test-slave-cr22 krb5kdc[3313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:55.329448 3326 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:42:55.330396 3326 webserver.cc:460] Webserver started at http://127.0.98.62:33285/ using document root <none> and password file <none>
I20260619 14:42:55.330942 3326 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:55.331032 3326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:55.331257 3326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:55.333343 3326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "b0679b1f80e34c289ce744b42a5f2e09"
format_stamp: "Formatted at 2026-06-19 14:42:55 on dist-test-slave-cr22"
server_key: "0300fe05f61b81055279662e20183067"
server_key_iv: "3fd76bb043aaa07598435f2610b5c746"
server_key_version: "encryptionkey@0"
I20260619 14:42:55.333957 3326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0679b1f80e34c289ce744b42a5f2e09"
format_stamp: "Formatted at 2026-06-19 14:42:55 on dist-test-slave-cr22"
server_key: "0300fe05f61b81055279662e20183067"
server_key_iv: "3fd76bb043aaa07598435f2610b5c746"
server_key_version: "encryptionkey@0"
I20260619 14:42:55.337728 3326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 14:42:55.340188 3341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:55.341392 3326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:42:55.341528 3326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "b0679b1f80e34c289ce744b42a5f2e09"
format_stamp: "Formatted at 2026-06-19 14:42:55 on dist-test-slave-cr22"
server_key: "0300fe05f61b81055279662e20183067"
server_key_iv: "3fd76bb043aaa07598435f2610b5c746"
server_key_version: "encryptionkey@0"
I20260619 14:42:55.341640 3326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:55.363417 3326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:55.366389 3326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:55.366596 3326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:55.373716 3326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40659
I20260619 14:42:55.374055 3393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40659 every 8 connection(s)
I20260619 14:42:55.374698 3326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:42:55.376439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 3326
I20260619 14:42:55.376566 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:42:55.376888 392 external_mini_cluster.cc:1428] Setting key 292ad42fdc31ab2f78534c040a321a4d
I20260619 14:42:55.378235 3394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:55.384209 3394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09: Bootstrap starting.
Jun 19 14:42:55 dist-test-slave-cr22 krb5kdc[3313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:55.386957 3394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:55.388018 3394 log.cc:824] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09: Log is configured to *not* fsync() on all Append() calls
I20260619 14:42:55.390223 3394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09: No bootstrap required, opened a new log
I20260619 14:42:55.392192 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:55.378309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41778 (local address 127.0.98.62:40659)
0619 14:42:55.378790 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:55.378804 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:55.378839 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:42:55.379569 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:55.379586 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:55.379979 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:55.380336 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:55.381133 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:55.382218 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:55.383115 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:55.383656 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:55.383825 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:55.386696 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:55.386701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:55.386714 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:55.386741 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:55.388995 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:55.389478 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:55.389484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:55.389498 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:55.389609 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:55.389916 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:55.389919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:55.389920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:55.390400 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:55.390599 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:55.390922 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 14:42:55.391211 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":160,"threads_started":1}
I20260619 14:42:55.393498 3394 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0679b1f80e34c289ce744b42a5f2e09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40659 } }
I20260619 14:42:55.393726 3394 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:55.393787 3394 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0679b1f80e34c289ce744b42a5f2e09, State: Initialized, Role: FOLLOWER
I20260619 14:42:55.394271 3394 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0679b1f80e34c289ce744b42a5f2e09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40659 } }
I20260619 14:42:55.394429 3394 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:42:55.394502 3394 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:42:55.394618 3394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:55.395676 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:42:55.395947 3394 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0679b1f80e34c289ce744b42a5f2e09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40659 } }
I20260619 14:42:55.396298 3394 leader_election.cc:302] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0679b1f80e34c289ce744b42a5f2e09; no voters:
I20260619 14:42:55.396683 3394 leader_election.cc:288] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:42:55.396762 3399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:42:55.396958 3399 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09 [term 1 LEADER]: Becoming Leader. State: Replica: b0679b1f80e34c289ce744b42a5f2e09, State: Running, Role: LEADER
I20260619 14:42:55.397330 3399 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0679b1f80e34c289ce744b42a5f2e09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40659 } }
I20260619 14:42:55.398013 3394 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:42:55.398744 3399 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0679b1f80e34c289ce744b42a5f2e09. Latest consensus state: current_term: 1 leader_uuid: "b0679b1f80e34c289ce744b42a5f2e09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0679b1f80e34c289ce744b42a5f2e09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40659 } } }
I20260619 14:42:55.398881 3399 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09 [sys.catalog]: This master's current role is: LEADER
I20260619 14:42:55.399144 3400 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0679b1f80e34c289ce744b42a5f2e09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0679b1f80e34c289ce744b42a5f2e09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40659 } } }
I20260619 14:42:55.399382 3400 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09 [sys.catalog]: This master's current role is: LEADER
I20260619 14:42:55.399375 3407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:42:55.403010 3407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:42:55.410210 3407 catalog_manager.cc:1347] Generated new cluster ID: ee57ef89921c4a079fc1cd8adb887a8c
I20260619 14:42:55.410292 3407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:42:55.416662 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:55.411003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41784 (local address 127.0.98.62:40659)
0619 14:42:55.411141 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:55.411146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:55.411163 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:42:55.411213 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:55.411216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:55.411267 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:55.411364 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:55.411985 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:55.412886 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:55.414033 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:55.414464 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:55.414585 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:55.415546 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:55.415550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:55.415553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:42:55.416147 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:55.416185 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:42:55.416207 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:55.416308 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:55.416419 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 14:42:55.416485 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:42:55.421967 3407 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:42:55.422972 3407 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:42:55.427829 3407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0679b1f80e34c289ce744b42a5f2e09: Generated new TSK 0
I20260619 14:42:55.428638 3407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:42:55.440872 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 3326
2026-06-19T14:42:55Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1425 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:42:55 dist-test-slave-cr22 krb5kdc[3427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:42:55 dist-test-slave-cr22 krb5kdc[3427](info): set up 2 sockets
Jun 19 14:42:55 dist-test-slave-cr22 krb5kdc[3427](info): commencing operation
krb5kdc: starting...
W20260619 14:42:56.500147 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:42:56 dist-test-slave-cr22 krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:42:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:42:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:42:56.630053 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39741
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:56.740471 3440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:56.740769 3440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:56.740818 3440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:56.744846 3440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:42:56.744910 3440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:56.744931 3440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:56.744948 3440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:42:56.744963 3440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:42:56.750741 3440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:56.751840 3440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:56.752811 3440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:56.762450 3445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:56.762569 3448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:56.762488 3446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:56.762724 3440 server_base.cc:1034] running on GCE node
I20260619 14:42:56.763643 3440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:56.765271 3440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:56.766467 3440 hybrid_clock.cc:648] HybridClock initialized: now 1781880176766426 us; error 64 us; skew 500 ppm
Jun 19 14:42:56 dist-test-slave-cr22 krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:56.769419 3440 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:42:56.770735 3440 webserver.cc:460] Webserver started at http://127.0.98.62:42567/ using document root <none> and password file <none>
I20260619 14:42:56.771536 3440 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:56.771633 3440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:56.771881 3440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:56.774024 3440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "27cb3ed8c970411a9de9901a8cb0aacb"
format_stamp: "Formatted at 2026-06-19 14:42:56 on dist-test-slave-cr22"
server_key: "3cdf62732b734f62098e8c963bd05f34"
server_key_iv: "30e72e7a86adc9667666064f536b9e19"
server_key_version: "encryptionkey@0"
I20260619 14:42:56.774581 3440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "27cb3ed8c970411a9de9901a8cb0aacb"
format_stamp: "Formatted at 2026-06-19 14:42:56 on dist-test-slave-cr22"
server_key: "3cdf62732b734f62098e8c963bd05f34"
server_key_iv: "30e72e7a86adc9667666064f536b9e19"
server_key_version: "encryptionkey@0"
I20260619 14:42:56.778647 3440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:42:56.781440 3455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:56.782795 3440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:42:56.782971 3440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "27cb3ed8c970411a9de9901a8cb0aacb"
format_stamp: "Formatted at 2026-06-19 14:42:56 on dist-test-slave-cr22"
server_key: "3cdf62732b734f62098e8c963bd05f34"
server_key_iv: "30e72e7a86adc9667666064f536b9e19"
server_key_version: "encryptionkey@0"
I20260619 14:42:56.783097 3440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:56.810550 3440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:56.814194 3440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:56.814428 3440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:56.822639 3440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39741
I20260619 14:42:56.822979 3507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39741 every 8 connection(s)
I20260619 14:42:56.823832 3440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:42:56.826716 3508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:56.826778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 3440
I20260619 14:42:56.826889 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:42:56.827181 392 external_mini_cluster.cc:1428] Setting key 16f548590159654823a4a6bc11fa751e
I20260619 14:42:56.833642 3508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb: Bootstrap starting.
I20260619 14:42:56.836337 3508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb: Neither blocks nor log segments found. Creating new log.
Jun 19 14:42:56 dist-test-slave-cr22 krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:56.837129 3508 log.cc:824] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb: Log is configured to *not* fsync() on all Append() calls
I20260619 14:42:56.839035 3508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb: No bootstrap required, opened a new log
I20260619 14:42:56.841913 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:56.828663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33346 (local address 127.0.98.62:39741)
0619 14:42:56.829102 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:56.829120 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:56.829160 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:42:56.829963 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:56.829985 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:56.830398 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:56.830758 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:56.831423 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:56.832925 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:56.833813 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:56.834347 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:56.834481 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:56.837387 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:56.837393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:56.837409 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:56.837441 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:56.839160 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:56.839680 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:56.839684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:56.839689 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:56.839791 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:56.840087 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:56.840089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:56.840090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:56.840498 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:56.840642 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:56.840935 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 14:42:56.841209 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":128,"threads_started":1}
I20260619 14:42:56.843116 3508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cb3ed8c970411a9de9901a8cb0aacb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39741 } }
I20260619 14:42:56.843381 3508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:56.843480 3508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27cb3ed8c970411a9de9901a8cb0aacb, State: Initialized, Role: FOLLOWER
I20260619 14:42:56.843916 3508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cb3ed8c970411a9de9901a8cb0aacb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39741 } }
I20260619 14:42:56.844084 3508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:42:56.844173 3508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:42:56.844301 3508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:56.845224 3508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cb3ed8c970411a9de9901a8cb0aacb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39741 } }
I20260619 14:42:56.845553 3508 leader_election.cc:302] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27cb3ed8c970411a9de9901a8cb0aacb; no voters:
I20260619 14:42:56.845856 3508 leader_election.cc:288] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:42:56.846006 3513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:42:56.846267 3513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb [term 1 LEADER]: Becoming Leader. State: Replica: 27cb3ed8c970411a9de9901a8cb0aacb, State: Running, Role: LEADER
I20260619 14:42:56.846583 3513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cb3ed8c970411a9de9901a8cb0aacb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39741 } }
I20260619 14:42:56.847055 3508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:42:56.848272 3515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27cb3ed8c970411a9de9901a8cb0aacb. Latest consensus state: current_term: 1 leader_uuid: "27cb3ed8c970411a9de9901a8cb0aacb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cb3ed8c970411a9de9901a8cb0aacb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39741 } } }
I20260619 14:42:56.848361 3514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27cb3ed8c970411a9de9901a8cb0aacb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cb3ed8c970411a9de9901a8cb0aacb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39741 } } }
I20260619 14:42:56.848448 3515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb [sys.catalog]: This master's current role is: LEADER
I20260619 14:42:56.848806 3514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb [sys.catalog]: This master's current role is: LEADER
I20260619 14:42:56.848903 3523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:42:56.851692 3523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:42:56.857223 3523 catalog_manager.cc:1347] Generated new cluster ID: 3035b7f579f14d7585006c9faf6ab00b
I20260619 14:42:56.857296 3523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:42:56.875286 3523 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:42:56.876583 3523 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:42:56.882314 3523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27cb3ed8c970411a9de9901a8cb0aacb: Generated new TSK 0
I20260619 14:42:56.883142 3523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:42:56.911077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39741
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:57.025907 3534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:57.026183 3534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:57.026227 3534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:57.030371 3534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:57.030447 3534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:57.030508 3534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:42:57.036408 3534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:57.037619 3534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:57.038611 3534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:57.049084 3542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:57.049299 3534 server_base.cc:1034] running on GCE node
W20260619 14:42:57.049255 3539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:57.049086 3540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:57.049903 3534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:57.050565 3534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:57.051731 3534 hybrid_clock.cc:648] HybridClock initialized: now 1781880177051712 us; error 34 us; skew 500 ppm
Jun 19 14:42:57 dist-test-slave-cr22 krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:57.054829 3534 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:42:57.055900 3534 webserver.cc:460] Webserver started at http://127.0.98.1:37433/ using document root <none> and password file <none>
I20260619 14:42:57.056485 3534 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:57.056578 3534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:57.056833 3534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:57.058930 3534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "c8b68e21b49044f887410c0fda2fbb7b"
format_stamp: "Formatted at 2026-06-19 14:42:57 on dist-test-slave-cr22"
server_key: "b7c56c9e5c5fe09e2226bf6ab9983b35"
server_key_iv: "d580a642213684c332dd2943627b9c3d"
server_key_version: "encryptionkey@0"
I20260619 14:42:57.059567 3534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c8b68e21b49044f887410c0fda2fbb7b"
format_stamp: "Formatted at 2026-06-19 14:42:57 on dist-test-slave-cr22"
server_key: "b7c56c9e5c5fe09e2226bf6ab9983b35"
server_key_iv: "d580a642213684c332dd2943627b9c3d"
server_key_version: "encryptionkey@0"
I20260619 14:42:57.064186 3534 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:42:57.067266 3549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:57.068939 3534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:42:57.069103 3534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "c8b68e21b49044f887410c0fda2fbb7b"
format_stamp: "Formatted at 2026-06-19 14:42:57 on dist-test-slave-cr22"
server_key: "b7c56c9e5c5fe09e2226bf6ab9983b35"
server_key_iv: "d580a642213684c332dd2943627b9c3d"
server_key_version: "encryptionkey@0"
I20260619 14:42:57.069236 3534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:57.094209 3534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:57.098169 3534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:57.098407 3534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:57.099080 3534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:42:57.100306 3534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:57.100399 3534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:57.100484 3534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:57.100535 3534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:57.110688 3534 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46673
I20260619 14:42:57.110766 3662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46673 every 8 connection(s)
I20260619 14:42:57.111737 3534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:42:57.118217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 3534
I20260619 14:42:57.118392 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:42:57.118681 392 external_mini_cluster.cc:1428] Setting key 9def46b47675cab4080c954093b2111f
Jun 19 14:42:57 dist-test-slave-cr22 krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:57.128597 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:57.113786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37441 (local address 127.0.98.62:39741)
0619 14:42:57.113942 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:57.113947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:57.114946 (+ 999us) server_negotiation.cc:407] Connection header received
0619 14:42:57.116459 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:57.116461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:57.116509 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:57.116586 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:57.117974 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:57.118775 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:57.120432 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:57.120793 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:57.120927 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:57.124276 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:57.124280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:57.124282 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:57.124307 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:57.125960 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:57.126694 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:57.126697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:57.126698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:57.126789 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:57.127230 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:57.127233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:57.127235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:57.127536 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:57.127620 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:57.128360 (+ 740us) server_negotiation.cc:299] Negotiation successful
0619 14:42:57.128475 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:42:57.129719 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:57.114192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39741 (local address 127.0.98.1:37441)
0619 14:42:57.114758 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:57.114823 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:57.116238 (+ 1415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:57.116795 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:57.116802 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:57.117226 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:57.117700 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:57.117712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:57.118988 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:57.118992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:57.120307 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:57.120318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:57.120976 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:57.120979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:57.121166 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:57.121901 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:57.121931 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:57.124082 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:57.126091 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:57.126100 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:57.126121 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:57.126543 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:57.126897 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:57.126902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:57.126905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:57.127103 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:57.127649 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:57.127657 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:57.128027 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 14:42:57.128255 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 14:42:57.128668 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":123,"threads_started":1}
I20260619 14:42:57.131093 3663 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39741
I20260619 14:42:57.131390 3663 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:42:57.131989 3663 heartbeater.cc:511] Master 127.0.98.62:39741 requested a full tablet report, sending...
I20260619 14:42:57.133625 3472 ts_manager.cc:194] Registered new tserver with Master: c8b68e21b49044f887410c0fda2fbb7b (127.0.98.1:46673)
I20260619 14:42:57.134848 3472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:42:57.152164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39741
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:57.272132 3668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:57.272395 3668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:57.272471 3668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:57.276773 3668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:57.276841 3668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:57.276901 3668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:42:57.282343 3668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:57.283317 3668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:57.284334 3668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:57.294812 3674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:57.294816 3673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:57.294926 3668 server_base.cc:1034] running on GCE node
W20260619 14:42:57.294885 3676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:57.295631 3668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:57.296252 3668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:57.297410 3668 hybrid_clock.cc:648] HybridClock initialized: now 1781880177297388 us; error 40 us; skew 500 ppm
Jun 19 14:42:57 dist-test-slave-cr22 krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:57.300220 3668 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:42:57.301038 3668 webserver.cc:460] Webserver started at http://127.0.98.2:42719/ using document root <none> and password file <none>
I20260619 14:42:57.301548 3668 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:57.301604 3668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:57.301815 3668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:57.303795 3668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "76a203c5c8a94900b4e37a2877fc011d"
format_stamp: "Formatted at 2026-06-19 14:42:57 on dist-test-slave-cr22"
server_key: "3b3390c52f4b2b0fc985a18190780cec"
server_key_iv: "8ca893bf0de77c4c73bce8cc8ac5c07b"
server_key_version: "encryptionkey@0"
I20260619 14:42:57.304288 3668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "76a203c5c8a94900b4e37a2877fc011d"
format_stamp: "Formatted at 2026-06-19 14:42:57 on dist-test-slave-cr22"
server_key: "3b3390c52f4b2b0fc985a18190780cec"
server_key_iv: "8ca893bf0de77c4c73bce8cc8ac5c07b"
server_key_version: "encryptionkey@0"
I20260619 14:42:57.307900 3668 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:42:57.310258 3683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:57.311622 3668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:42:57.311748 3668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "76a203c5c8a94900b4e37a2877fc011d"
format_stamp: "Formatted at 2026-06-19 14:42:57 on dist-test-slave-cr22"
server_key: "3b3390c52f4b2b0fc985a18190780cec"
server_key_iv: "8ca893bf0de77c4c73bce8cc8ac5c07b"
server_key_version: "encryptionkey@0"
I20260619 14:42:57.311864 3668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:57.326434 3668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:57.329630 3668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:57.329833 3668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:57.330386 3668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:42:57.331157 3668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:57.331231 3668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:57.331313 3668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:57.331365 3668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:57.339762 3668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39223
I20260619 14:42:57.339792 3796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39223 every 8 connection(s)
I20260619 14:42:57.340619 3668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:42:57.349623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 3668
I20260619 14:42:57.349743 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:42:57.350102 392 external_mini_cluster.cc:1428] Setting key 1119baef05610125e3af8babba5226c6
Jun 19 14:42:57 dist-test-slave-cr22 krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:57.355499 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:57.342493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35285 (local address 127.0.98.62:39741)
0619 14:42:57.342628 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:57.342632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:57.343499 (+ 867us) server_negotiation.cc:407] Connection header received
0619 14:42:57.344945 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:57.344948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:57.345008 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:57.345114 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:57.346192 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:57.346723 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:57.347961 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:57.348242 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:57.348383 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:57.351064 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:57.351068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:57.351071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:57.351096 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:57.352933 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:57.353574 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:57.353579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:57.353582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:57.353681 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:57.354138 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:57.354145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:57.354146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:57.354419 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:57.354531 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:57.355080 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 14:42:57.355221 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:42:57.356374 3799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:57.342909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39741 (local address 127.0.98.2:35285)
0619 14:42:57.343343 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:57.343383 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:57.344667 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:57.345275 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:57.345281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:57.345611 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:57.346055 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:57.346066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:57.346859 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:57.346865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:57.347835 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:57.347844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:57.348418 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:57.348422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:57.348582 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:57.349088 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:57.349104 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:57.350895 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:57.353048 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:57.353052 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:57.353063 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:57.353442 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:57.353812 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:57.353815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:57.353817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:57.354010 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:57.354560 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:57.354566 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:57.354820 (+ 254us) client_negotiation.cc:769] Sending connection context
0619 14:42:57.355004 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 14:42:57.355281 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":115,"threads_started":1}
I20260619 14:42:57.357630 3797 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39741
I20260619 14:42:57.357892 3797 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:57.358484 3797 heartbeater.cc:511] Master 127.0.98.62:39741 requested a full tablet report, sending...
I20260619 14:42:57.359679 3472 ts_manager.cc:194] Registered new tserver with Master: 76a203c5c8a94900b4e37a2877fc011d (127.0.98.2:39223)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:42:57.360529 3472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:42:57.379140 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39741
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:57.489202 3802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:57.489485 3802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:57.489529 3802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:57.493970 3802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:57.494035 3802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:57.494096 3802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:42:57.499656 3802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:57.500756 3802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:57.501776 3802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:57.511863 3810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:57.511794 3808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:57.511794 3807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:57.511909 3802 server_base.cc:1034] running on GCE node
I20260619 14:42:57.512583 3802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:57.513173 3802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:57.514317 3802 hybrid_clock.cc:648] HybridClock initialized: now 1781880177514296 us; error 34 us; skew 500 ppm
Jun 19 14:42:57 dist-test-slave-cr22 krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:57.517333 3802 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:42:57.518307 3802 webserver.cc:460] Webserver started at http://127.0.98.3:42453/ using document root <none> and password file <none>
I20260619 14:42:57.518878 3802 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:57.518939 3802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:57.519162 3802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:57.521286 3802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "87f0e2aac5d54366a8d484e1223692f0"
format_stamp: "Formatted at 2026-06-19 14:42:57 on dist-test-slave-cr22"
server_key: "637e9a05c961373b1e8801c5f80e0df6"
server_key_iv: "1b0b7118fda5076debfdd12ee912303c"
server_key_version: "encryptionkey@0"
I20260619 14:42:57.521873 3802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "87f0e2aac5d54366a8d484e1223692f0"
format_stamp: "Formatted at 2026-06-19 14:42:57 on dist-test-slave-cr22"
server_key: "637e9a05c961373b1e8801c5f80e0df6"
server_key_iv: "1b0b7118fda5076debfdd12ee912303c"
server_key_version: "encryptionkey@0"
I20260619 14:42:57.525815 3802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:42:57.528625 3817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:57.530028 3802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:42:57.530216 3802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "87f0e2aac5d54366a8d484e1223692f0"
format_stamp: "Formatted at 2026-06-19 14:42:57 on dist-test-slave-cr22"
server_key: "637e9a05c961373b1e8801c5f80e0df6"
server_key_iv: "1b0b7118fda5076debfdd12ee912303c"
server_key_version: "encryptionkey@0"
I20260619 14:42:57.530359 3802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:57.545650 3802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:57.549399 3802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:57.549634 3802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:57.550339 3802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:42:57.551276 3802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:57.551342 3802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:57.551532 3802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:57.551591 3802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:57.561082 3802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38649
I20260619 14:42:57.561100 3930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38649 every 8 connection(s)
I20260619 14:42:57.562098 3802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:42:57.565403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 3802
I20260619 14:42:57.565528 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:42:57.565872 392 external_mini_cluster.cc:1428] Setting key 4954b02fe34b1d1134a22befd22427dc
Jun 19 14:42:57 dist-test-slave-cr22 krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:57.577111 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:57.564011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33193 (local address 127.0.98.62:39741)
0619 14:42:57.564149 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:57.564153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:57.565044 (+ 891us) server_negotiation.cc:407] Connection header received
0619 14:42:57.566938 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:57.566943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:57.567037 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:57.567124 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:57.568504 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:57.569104 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:57.570222 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:57.570538 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:57.570658 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:57.573579 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:57.573582 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:57.573584 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:57.573607 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:57.575090 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:57.575641 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:57.575643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:57.575645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:57.575716 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:57.576058 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:57.576060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:57.576062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:57.576226 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:57.576297 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:57.576892 (+ 595us) server_negotiation.cc:299] Negotiation successful
0619 14:42:57.576990 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:42:57.577880 3933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:57.564413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39741 (local address 127.0.98.3:33193)
0619 14:42:57.564889 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:57.564933 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:57.566696 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:57.567349 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:57.567356 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:57.567773 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:57.568355 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:57.568370 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:57.569227 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:57.569230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:57.570091 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:57.570099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:57.570709 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:57.570716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:57.571032 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:57.571766 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:57.571783 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:57.573414 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:57.575193 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:57.575198 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:57.575209 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:57.575552 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:57.575822 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:57.575825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:57.575827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:57.575962 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:57.576312 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:57.576317 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:57.576605 (+ 288us) client_negotiation.cc:769] Sending connection context
0619 14:42:57.576777 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 14:42:57.577029 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":112,"threads_started":1}
I20260619 14:42:57.579177 3931 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39741
I20260619 14:42:57.579420 3931 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:57.580003 3931 heartbeater.cc:511] Master 127.0.98.62:39741 requested a full tablet report, sending...
I20260619 14:42:57.581192 3472 ts_manager.cc:194] Registered new tserver with Master: 87f0e2aac5d54366a8d484e1223692f0 (127.0.98.3:38649)
I20260619 14:42:57.581772 3472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33193
I20260619 14:42:57.592061 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:42:57 dist-test-slave-cr22 krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880177, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 14:42:57 dist-test-slave-cr22 krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880177, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:57.618335 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:57.609527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33358 (local address 127.0.98.62:39741)
0619 14:42:57.609716 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:57.609721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:57.609908 (+ 187us) server_negotiation.cc:407] Connection header received
0619 14:42:57.610076 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:57.610079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:57.610131 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:57.610219 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:57.610921 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:57.611506 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:57.612352 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:57.612585 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:57.612693 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:57.615202 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:57.615206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:57.615208 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:57.615235 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:57.616677 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:57.617099 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:57.617109 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:57.617111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:57.617182 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:57.617448 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:57.617453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:57.617455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:57.617666 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:57.617799 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:57.618046 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 14:42:57.618162 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 14:42:57.618709 3472 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33358
I20260619 14:42:57.620100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 3534
I20260619 14:42:57.626096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 3668
I20260619 14:42:57.631362 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 3802
I20260619 14:42:57.636914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 3440
2026-06-19T14:42:57Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2197 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:42:57 dist-test-slave-cr22 krb5kdc[3943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:42:57 dist-test-slave-cr22 krb5kdc[3943](info): set up 2 sockets
Jun 19 14:42:57 dist-test-slave-cr22 krb5kdc[3943](info): commencing operation
krb5kdc: starting...
W20260619 14:42:58.728607 392 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:42:58 dist-test-slave-cr22 krb5kdc[3943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:42:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:42:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:42:58.859092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39639
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:58.967250 3956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:58.967597 3956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:58.967680 3956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:58.971752 3956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:42:58.971871 3956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:58.971911 3956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:58.971967 3956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:42:58.972003 3956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:42:58.977303 3956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:58.978400 3956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:58.979681 3956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:58.988270 3964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:58.988320 3961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:58.988361 3962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:58.988505 3956 server_base.cc:1034] running on GCE node
I20260619 14:42:58.989094 3956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:58.990213 3956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:58.991379 3956 hybrid_clock.cc:648] HybridClock initialized: now 1781880178991362 us; error 31 us; skew 500 ppm
Jun 19 14:42:58 dist-test-slave-cr22 krb5kdc[3943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:58.994144 3956 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:42:58.995152 3956 webserver.cc:460] Webserver started at http://127.0.98.62:42751/ using document root <none> and password file <none>
I20260619 14:42:58.995757 3956 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:58.995826 3956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:58.996093 3956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:58.998188 3956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "6fcc7359f0fd47998b6876a8a4efc0d3"
format_stamp: "Formatted at 2026-06-19 14:42:58 on dist-test-slave-cr22"
server_key: "5bb58e726551cec9b7f4ca4f5dd48e6e"
server_key_iv: "fab97266ea33902da450845ae45dc599"
server_key_version: "encryptionkey@0"
I20260619 14:42:58.998759 3956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "6fcc7359f0fd47998b6876a8a4efc0d3"
format_stamp: "Formatted at 2026-06-19 14:42:58 on dist-test-slave-cr22"
server_key: "5bb58e726551cec9b7f4ca4f5dd48e6e"
server_key_iv: "fab97266ea33902da450845ae45dc599"
server_key_version: "encryptionkey@0"
I20260619 14:42:59.002581 3956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 14:42:59.005259 3971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:59.006493 3956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:42:59.006629 3956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "6fcc7359f0fd47998b6876a8a4efc0d3"
format_stamp: "Formatted at 2026-06-19 14:42:58 on dist-test-slave-cr22"
server_key: "5bb58e726551cec9b7f4ca4f5dd48e6e"
server_key_iv: "fab97266ea33902da450845ae45dc599"
server_key_version: "encryptionkey@0"
I20260619 14:42:59.006752 3956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:59.036283 3956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:59.040164 3956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:59.040426 3956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:59.048380 3956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39639
I20260619 14:42:59.048442 4023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39639 every 8 connection(s)
I20260619 14:42:59.049393 3956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:42:59.052160 4024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:42:59.055678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 3956
I20260619 14:42:59.055833 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:42:59.056109 392 external_mini_cluster.cc:1428] Setting key 719fa4584f7be4e39ddee06577fea444
I20260619 14:42:59.058249 4024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3: Bootstrap starting.
I20260619 14:42:59.060489 4024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3: Neither blocks nor log segments found. Creating new log.
I20260619 14:42:59.061398 4024 log.cc:824] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3: Log is configured to *not* fsync() on all Append() calls
I20260619 14:42:59.063514 4024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3: No bootstrap required, opened a new log
Jun 19 14:42:59 dist-test-slave-cr22 krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:59.067421 4024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fcc7359f0fd47998b6876a8a4efc0d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39639 } }
I20260619 14:42:59.067790 4024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:42:59.067852 4024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fcc7359f0fd47998b6876a8a4efc0d3, State: Initialized, Role: FOLLOWER
I20260619 14:42:59.068373 4024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fcc7359f0fd47998b6876a8a4efc0d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39639 } }
I20260619 14:42:59.068524 4024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:42:59.068588 4024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:42:59.068689 4024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:42:59.069834 4024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fcc7359f0fd47998b6876a8a4efc0d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39639 } }
I20260619 14:42:59.070183 4024 leader_election.cc:302] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fcc7359f0fd47998b6876a8a4efc0d3; no voters:
I20260619 14:42:59.070513 4024 leader_election.cc:288] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:42:59.070662 4029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:42:59.070997 4029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3 [term 1 LEADER]: Becoming Leader. State: Replica: 6fcc7359f0fd47998b6876a8a4efc0d3, State: Running, Role: LEADER
I20260619 14:42:59.071324 4029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fcc7359f0fd47998b6876a8a4efc0d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39639 } }
I20260619 14:42:59.071834 4024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:42:59.072552 4029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fcc7359f0fd47998b6876a8a4efc0d3. Latest consensus state: current_term: 1 leader_uuid: "6fcc7359f0fd47998b6876a8a4efc0d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fcc7359f0fd47998b6876a8a4efc0d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39639 } } }
I20260619 14:42:59.072774 4029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:42:59.072791 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:59.057741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.0.98.62:39639)
0619 14:42:59.058218 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:59.058234 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:59.058271 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:42:59.058998 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:59.059015 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:59.059362 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:59.059899 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:59.060729 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:59.062196 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:59.063046 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:59.063723 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:59.063896 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:59.066647 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:59.066653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:59.066679 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:59.066719 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:59.068979 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:59.069424 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:59.069432 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:59.069439 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:59.069586 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:59.070042 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:59.070046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:59.070048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:59.070584 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:59.070770 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:59.071509 (+ 739us) server_negotiation.cc:299] Negotiation successful
0619 14:42:59.071801 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":132,"threads_started":1}
I20260619 14:42:59.072992 4030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fcc7359f0fd47998b6876a8a4efc0d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fcc7359f0fd47998b6876a8a4efc0d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39639 } } }
I20260619 14:42:59.073266 4038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:42:59.073292 4030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:42:59.076491 4038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:42:59.082064 4038 catalog_manager.cc:1347] Generated new cluster ID: 6d52bf36fef444edba22ac758e7facc5
I20260619 14:42:59.082136 4038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:42:59.096334 4038 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:42:59.097406 4038 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:42:59.101770 4038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fcc7359f0fd47998b6876a8a4efc0d3: Generated new TSK 0
I20260619 14:42:59.102515 4038 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:42:59.138813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39639
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:59.252004 4050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:59.252279 4050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:59.252362 4050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:59.256587 4050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:59.256667 4050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:59.256767 4050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:42:59.262179 4050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:59.263350 4050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:59.264436 4050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:59.274336 4055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:59.274438 4058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:59.274418 4050 server_base.cc:1034] running on GCE node
W20260619 14:42:59.274338 4056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:59.275070 4050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:59.275847 4050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:59.277060 4050 hybrid_clock.cc:648] HybridClock initialized: now 1781880179277041 us; error 31 us; skew 500 ppm
Jun 19 14:42:59 dist-test-slave-cr22 krb5kdc[3943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:59.279999 4050 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:42:59.280850 4050 webserver.cc:460] Webserver started at http://127.0.98.1:36177/ using document root <none> and password file <none>
I20260619 14:42:59.281426 4050 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:59.281517 4050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:59.281747 4050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:59.283962 4050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "b80d1dc0773342e5807c38254a77385b"
format_stamp: "Formatted at 2026-06-19 14:42:59 on dist-test-slave-cr22"
server_key: "7b6fd451e7202040c7924a8e6fe156b4"
server_key_iv: "b1cb05582c4ffc15eb56f34cf3ec6bfb"
server_key_version: "encryptionkey@0"
I20260619 14:42:59.284528 4050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b80d1dc0773342e5807c38254a77385b"
format_stamp: "Formatted at 2026-06-19 14:42:59 on dist-test-slave-cr22"
server_key: "7b6fd451e7202040c7924a8e6fe156b4"
server_key_iv: "b1cb05582c4ffc15eb56f34cf3ec6bfb"
server_key_version: "encryptionkey@0"
I20260619 14:42:59.288326 4050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:42:59.290848 4065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:59.292359 4050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:42:59.292475 4050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "b80d1dc0773342e5807c38254a77385b"
format_stamp: "Formatted at 2026-06-19 14:42:59 on dist-test-slave-cr22"
server_key: "7b6fd451e7202040c7924a8e6fe156b4"
server_key_iv: "b1cb05582c4ffc15eb56f34cf3ec6bfb"
server_key_version: "encryptionkey@0"
I20260619 14:42:59.292599 4050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:59.304243 4050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:59.307121 4050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:59.307276 4050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:59.307862 4050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:42:59.308604 4050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:59.308650 4050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:59.308687 4050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:59.308702 4050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:59.317642 4050 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39649
I20260619 14:42:59.317715 4178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39649 every 8 connection(s)
I20260619 14:42:59.318562 4050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:42:59.325418 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 4050
I20260619 14:42:59.325549 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:42:59.325848 392 external_mini_cluster.cc:1428] Setting key 5145fe7bcd0a0a6aedb860a445cb7c9e
Jun 19 14:42:59 dist-test-slave-cr22 krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:59.333016 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:59.320352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54435 (local address 127.0.98.62:39639)
0619 14:42:59.320508 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:59.320513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:59.321302 (+ 789us) server_negotiation.cc:407] Connection header received
0619 14:42:59.322621 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:59.322624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:59.322668 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:59.322742 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:59.323956 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:59.324519 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:59.326015 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:59.326282 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:59.326400 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:59.329157 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:59.329160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:59.329162 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:59.329184 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:59.330881 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:59.331380 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:59.331386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:59.331388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:59.331523 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:59.331919 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:59.331922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:59.331923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:59.332091 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:59.332201 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:59.332708 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 14:42:59.332844 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:42:59.333868 4181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:59.320731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39639 (local address 127.0.98.1:54435)
0619 14:42:59.321152 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:59.321195 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:59.322428 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:59.322956 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:59.322962 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:59.323301 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:59.323788 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:59.323801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:59.324653 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:59.324656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:59.325888 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:59.325898 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:59.326468 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:59.326472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:59.326656 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:59.327268 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:59.327283 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:59.328973 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:59.330997 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:59.331001 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:59.331013 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:59.331259 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:59.331618 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:59.331621 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:59.331623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:59.331759 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:59.332197 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:59.332203 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:59.332476 (+ 273us) client_negotiation.cc:769] Sending connection context
0619 14:42:59.332668 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 14:42:59.332982 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:42:59.335194 4179 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39639
I20260619 14:42:59.335474 4179 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:59.335940 4179 heartbeater.cc:511] Master 127.0.98.62:39639 requested a full tablet report, sending...
I20260619 14:42:59.337404 3988 ts_manager.cc:194] Registered new tserver with Master: b80d1dc0773342e5807c38254a77385b (127.0.98.1:39649)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:42:59.338549 3988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:42:59.354807 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39639
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:59.462890 4184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:59.463158 4184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:59.463201 4184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:59.467199 4184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:59.467263 4184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:59.467321 4184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:42:59.472718 4184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:59.473824 4184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:59.474768 4184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:59.484715 4192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:59.484673 4189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:59.484860 4190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:59.485117 4184 server_base.cc:1034] running on GCE node
I20260619 14:42:59.485617 4184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:59.486292 4184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:59.487530 4184 hybrid_clock.cc:648] HybridClock initialized: now 1781880179487506 us; error 26 us; skew 500 ppm
Jun 19 14:42:59 dist-test-slave-cr22 krb5kdc[3943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:59.490711 4184 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:42:59.491868 4184 webserver.cc:460] Webserver started at http://127.0.98.2:32999/ using document root <none> and password file <none>
I20260619 14:42:59.492480 4184 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:59.492547 4184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:59.492782 4184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:59.495030 4184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "b7e29ecccdf84e0c9bbd534d6b7f5be7"
format_stamp: "Formatted at 2026-06-19 14:42:59 on dist-test-slave-cr22"
server_key: "98ea5558a02dd6b533211cc8eff39dfe"
server_key_iv: "c37b1750b4258de1874afe84202642be"
server_key_version: "encryptionkey@0"
I20260619 14:42:59.495934 4184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b7e29ecccdf84e0c9bbd534d6b7f5be7"
format_stamp: "Formatted at 2026-06-19 14:42:59 on dist-test-slave-cr22"
server_key: "98ea5558a02dd6b533211cc8eff39dfe"
server_key_iv: "c37b1750b4258de1874afe84202642be"
server_key_version: "encryptionkey@0"
I20260619 14:42:59.500458 4184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:42:59.503273 4199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:59.504932 4184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:42:59.505105 4184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "b7e29ecccdf84e0c9bbd534d6b7f5be7"
format_stamp: "Formatted at 2026-06-19 14:42:59 on dist-test-slave-cr22"
server_key: "98ea5558a02dd6b533211cc8eff39dfe"
server_key_iv: "c37b1750b4258de1874afe84202642be"
server_key_version: "encryptionkey@0"
I20260619 14:42:59.505246 4184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:59.517990 4184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:59.521257 4184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:59.521484 4184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:59.522076 4184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:42:59.522917 4184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:59.522989 4184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:59.523061 4184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:59.523114 4184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:59.532236 4184 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41707
I20260619 14:42:59.532287 4312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41707 every 8 connection(s)
I20260619 14:42:59.533145 4184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:42:59.541718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 4184
I20260619 14:42:59.541867 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:42:59.542162 392 external_mini_cluster.cc:1428] Setting key b2c07f728a07fc9f190b36e2c5d9b7d4
Jun 19 14:42:59 dist-test-slave-cr22 krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:59.548429 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:59.535111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38799 (local address 127.0.98.62:39639)
0619 14:42:59.535253 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:59.535257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:59.536136 (+ 879us) server_negotiation.cc:407] Connection header received
0619 14:42:59.537552 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:59.537556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:59.537618 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:59.537697 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:59.538870 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:59.539418 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:59.540611 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:59.540847 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:59.540960 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:59.543947 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:59.543951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:59.543954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:59.543983 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:59.545785 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:59.546531 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:59.546534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:59.546535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:59.546612 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:59.547155 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:59.547158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:59.547159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:59.547340 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:59.547420 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:59.548113 (+ 693us) server_negotiation.cc:299] Negotiation successful
0619 14:42:59.548226 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:42:59.549572 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:59.535382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39639 (local address 127.0.98.2:38799)
0619 14:42:59.535965 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:59.536007 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:59.537326 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:59.537864 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:59.537871 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:59.538224 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:59.538704 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:59.538717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:59.539592 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:59.539598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:59.540474 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:59.540483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:59.541000 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:59.541004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:59.541170 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:59.542009 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:59.542025 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:59.543755 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:59.546012 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:59.546020 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:59.546040 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:59.546422 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:59.546773 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:59.546778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:59.546781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:59.547045 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:59.547480 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:59.547490 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:59.547837 (+ 347us) client_negotiation.cc:769] Sending connection context
0619 14:42:59.548057 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 14:42:59.548431 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":119,"threads_started":1}
I20260619 14:42:59.550832 4313 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39639
I20260619 14:42:59.551110 4313 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:59.551745 4313 heartbeater.cc:511] Master 127.0.98.62:39639 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:42:59.552934 3988 ts_manager.cc:194] Registered new tserver with Master: b7e29ecccdf84e0c9bbd534d6b7f5be7 (127.0.98.2:41707)
I20260619 14:42:59.553534 3988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:42:59.575074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39639
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:42:59.685662 4318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:42:59.685915 4318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:42:59.685959 4318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:42:59.690006 4318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:42:59.690071 4318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:42:59.690171 4318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:42:59.695410 4318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:42:59.696516 4318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:42:59.697602 4318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:42:59.707283 4326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:59.707360 4318 server_base.cc:1034] running on GCE node
W20260619 14:42:59.707245 4323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:42:59.707403 4324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:42:59.707985 4318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:42:59.708595 4318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:42:59.709911 4318 hybrid_clock.cc:648] HybridClock initialized: now 1781880179709876 us; error 45 us; skew 500 ppm
Jun 19 14:42:59 dist-test-slave-cr22 krb5kdc[3943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:42:59.712690 4318 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:42:59.713568 4318 webserver.cc:460] Webserver started at http://127.0.98.3:43539/ using document root <none> and password file <none>
I20260619 14:42:59.714125 4318 fs_manager.cc:359] Metadata directory not provided
I20260619 14:42:59.714216 4318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:42:59.714464 4318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:42:59.716522 4318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "eac9fbc2721646de8d492b97292adcc1"
format_stamp: "Formatted at 2026-06-19 14:42:59 on dist-test-slave-cr22"
server_key: "eaff214c2534b81c1334cadb3af28a6e"
server_key_iv: "e063913712c6dfcf4349e65e2a13a34b"
server_key_version: "encryptionkey@0"
I20260619 14:42:59.717087 4318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eac9fbc2721646de8d492b97292adcc1"
format_stamp: "Formatted at 2026-06-19 14:42:59 on dist-test-slave-cr22"
server_key: "eaff214c2534b81c1334cadb3af28a6e"
server_key_iv: "e063913712c6dfcf4349e65e2a13a34b"
server_key_version: "encryptionkey@0"
I20260619 14:42:59.720796 4318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:42:59.723275 4333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:59.724555 4318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:42:59.724689 4318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "eac9fbc2721646de8d492b97292adcc1"
format_stamp: "Formatted at 2026-06-19 14:42:59 on dist-test-slave-cr22"
server_key: "eaff214c2534b81c1334cadb3af28a6e"
server_key_iv: "e063913712c6dfcf4349e65e2a13a34b"
server_key_version: "encryptionkey@0"
I20260619 14:42:59.724803 4318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:42:59.737744 4318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:42:59.741011 4318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:42:59.741223 4318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:42:59.741866 4318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:42:59.742704 4318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:42:59.742776 4318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:59.742847 4318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:42:59.742899 4318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:42:59.752569 4318 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33897
I20260619 14:42:59.752588 4446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33897 every 8 connection(s)
I20260619 14:42:59.753448 4318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:42:59.762161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 4318
I20260619 14:42:59.762310 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:42:59.762583 392 external_mini_cluster.cc:1428] Setting key c0d50b660f1e9236391ee0f110d8a044
Jun 19 14:42:59 dist-test-slave-cr22 krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:59.768417 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:59.755383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49419 (local address 127.0.98.62:39639)
0619 14:42:59.755570 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:59.755574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:59.756359 (+ 785us) server_negotiation.cc:407] Connection header received
0619 14:42:59.757879 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:59.757883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:59.757943 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:59.758048 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:59.759257 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:59.759877 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:59.761025 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:59.761265 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:59.761377 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:59.764502 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:59.764506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:59.764508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:59.764531 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:59.765972 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:59.766554 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:59.766557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:59.766558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:59.766632 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:59.767045 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:59.767048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:59.767049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:59.767269 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:59.767364 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:59.768168 (+ 804us) server_negotiation.cc:299] Negotiation successful
0619 14:42:59.768284 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 14:42:59.769296 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:59.755734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39639 (local address 127.0.98.3:49419)
0619 14:42:59.756200 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 14:42:59.756242 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:42:59.757666 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:42:59.758214 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:42:59.758221 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:42:59.758632 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:42:59.759095 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:59.759107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:59.760004 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:59.760010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:59.760896 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:42:59.760905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:42:59.761416 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:42:59.761420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:42:59.761583 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:59.762287 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:42:59.762324 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:42:59.764308 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:42:59.766102 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:59.766108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:59.766121 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:59.766453 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:59.766765 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:42:59.766768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:42:59.766769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:42:59.766915 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:42:59.767627 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:42:59.767635 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:42:59.767919 (+ 284us) client_negotiation.cc:769] Sending connection context
0619 14:42:59.768096 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 14:42:59.768377 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":105,"threads_started":1}
I20260619 14:42:59.770416 4447 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39639
I20260619 14:42:59.770629 4447 heartbeater.cc:464] Registering TS with master...
I20260619 14:42:59.771126 4447 heartbeater.cc:511] Master 127.0.98.62:39639 requested a full tablet report, sending...
I20260619 14:42:59.772399 3988 ts_manager.cc:194] Registered new tserver with Master: eac9fbc2721646de8d492b97292adcc1 (127.0.98.3:33897)
I20260619 14:42:59.773080 3988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49419
I20260619 14:42:59.776618 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:42:59 dist-test-slave-cr22 krb5kdc[3943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:42:59 dist-test-slave-cr22 krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:42:59.803297 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:59.793079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.0.98.1:39649)
0619 14:42:59.793389 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:59.793407 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:59.793438 (+ 31us) server_negotiation.cc:407] Connection header received
0619 14:42:59.793514 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:59.793520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:59.793720 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:59.793883 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:59.794627 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:59.795559 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:59.796465 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:59.796955 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:59.797047 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:59.799668 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:59.799678 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:59.799697 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:59.799732 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:59.801425 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:59.801879 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:59.801884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:59.801885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:59.801961 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:59.802253 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:59.802256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:59.802257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:59.802557 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:59.802734 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:59.802913 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 14:42:59.803094 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
Jun 19 14:42:59 dist-test-slave-cr22 krb5kdc[3943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:42:59.815151 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:42:59.806211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40438 (local address 127.0.98.62:39639)
0619 14:42:59.806388 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 14:42:59.806391 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:42:59.806405 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:42:59.806441 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:42:59.806443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:42:59.806481 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:42:59.806546 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:42:59.807144 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:59.807853 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:59.808603 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:42:59.808875 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:42:59.809030 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:42:59.811945 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:42:59.811949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:42:59.811951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:42:59.811974 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:42:59.813397 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:59.813839 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:59.813843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:59.813844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:59.813924 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:42:59.814202 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:42:59.814205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:42:59.814207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:42:59.814433 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:42:59.814591 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 14:42:59.814904 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 14:42:59.815014 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 14:42:59.815591 3988 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40438
I20260619 14:42:59.816215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 4050
I20260619 14:42:59.822299 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 4184
I20260619 14:42:59.827746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 4318
I20260619 14:42:59.832974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 3956
2026-06-19T14:42:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2196 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:42:59 dist-test-slave-cr22 krb5kdc[4459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:42:59 dist-test-slave-cr22 krb5kdc[4459](info): set up 2 sockets
Jun 19 14:42:59 dist-test-slave-cr22 krb5kdc[4459](info): commencing operation
krb5kdc: starting...
W20260619 14:43:00.935647 392 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:43:01 dist-test-slave-cr22 krb5kdc[4459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:43:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:43:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:43:01.062788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33549
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:01.175643 4472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:01.175980 4472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:01.176064 4472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:01.180330 4472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:43:01.180411 4472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:01.180449 4472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:01.180505 4472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:43:01.180543 4472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:43:01.185859 4472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:01.186944 4472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:01.188051 4472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:43:01.196918 4472 server_base.cc:1034] running on GCE node
W20260619 14:43:01.196945 4478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:01.196974 4477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:01.197094 4480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:01.197593 4472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:01.198671 4472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:01.199882 4472 hybrid_clock.cc:648] HybridClock initialized: now 1781880181199845 us; error 53 us; skew 500 ppm
Jun 19 14:43:01 dist-test-slave-cr22 krb5kdc[4459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:01.202723 4472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:43:01.203516 4472 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:01.203606 4472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:01.203833 4472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:01.205832 4472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "2cf0ee81227344c6986199dbd922970b"
format_stamp: "Formatted at 2026-06-19 14:43:01 on dist-test-slave-cr22"
server_key: "074ea3da0372d9d79c80abd605825121"
server_key_iv: "976d35fc78f9d8b8a7b51b2b936c45a5"
server_key_version: "encryptionkey@0"
I20260619 14:43:01.206387 4472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "2cf0ee81227344c6986199dbd922970b"
format_stamp: "Formatted at 2026-06-19 14:43:01 on dist-test-slave-cr22"
server_key: "074ea3da0372d9d79c80abd605825121"
server_key_iv: "976d35fc78f9d8b8a7b51b2b936c45a5"
server_key_version: "encryptionkey@0"
I20260619 14:43:01.210075 4472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:43:01.212566 4486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:01.213855 4472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:43:01.214007 4472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "2cf0ee81227344c6986199dbd922970b"
format_stamp: "Formatted at 2026-06-19 14:43:01 on dist-test-slave-cr22"
server_key: "074ea3da0372d9d79c80abd605825121"
server_key_iv: "976d35fc78f9d8b8a7b51b2b936c45a5"
server_key_version: "encryptionkey@0"
I20260619 14:43:01.214124 4472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:01.228904 4472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:01.232183 4472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:01.232420 4472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:01.239905 4472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33549
I20260619 14:43:01.239913 4538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33549 every 8 connection(s)
I20260619 14:43:01.240571 4472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:43:01.243503 4539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:01.249624 4539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b: Bootstrap starting.
I20260619 14:43:01.249866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 4472
I20260619 14:43:01.250000 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:43:01.250309 392 external_mini_cluster.cc:1428] Setting key 2d6489f02958f3fdb6aa81fc2fa87b0b
I20260619 14:43:01.252701 4539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:01.253502 4539 log.cc:824] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:01.255527 4539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b: No bootstrap required, opened a new log
Jun 19 14:43:01 dist-test-slave-cr22 krb5kdc[4459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:43:01.259480 4539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf0ee81227344c6986199dbd922970b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33549 } }
I20260619 14:43:01.259789 4539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:01.259868 4539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf0ee81227344c6986199dbd922970b, State: Initialized, Role: FOLLOWER
I20260619 14:43:01.260489 4539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf0ee81227344c6986199dbd922970b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33549 } }
I20260619 14:43:01.260722 4544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:43:01.260886 4544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf0ee81227344c6986199dbd922970b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33549 } }
I20260619 14:43:01.260700 4539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:43:01.261283 4539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:43:01.261279 4544 leader_election.cc:302] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cf0ee81227344c6986199dbd922970b; no voters:
I20260619 14:43:01.261418 4539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:01.261679 4544 leader_election.cc:288] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260619 14:43:01.262501 4539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf0ee81227344c6986199dbd922970b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33549 } }
I20260619 14:43:01.262658 4539 leader_election.cc:302] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cf0ee81227344c6986199dbd922970b; no voters:
I20260619 14:43:01.262691 4545 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 14:43:01.262775 4539 leader_election.cc:288] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:43:01.262953 4545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:01.263094 4545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [term 1 LEADER]: Becoming Leader. State: Replica: 2cf0ee81227344c6986199dbd922970b, State: Running, Role: LEADER
I20260619 14:43:01.263430 4545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf0ee81227344c6986199dbd922970b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33549 } }
I20260619 14:43:01.264154 4539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:43:01.265563 4544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cf0ee81227344c6986199dbd922970b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf0ee81227344c6986199dbd922970b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33549 } } }
I20260619 14:43:01.265954 4544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:01.265865 4546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cf0ee81227344c6986199dbd922970b. Latest consensus state: current_term: 1 leader_uuid: "2cf0ee81227344c6986199dbd922970b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cf0ee81227344c6986199dbd922970b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33549 } } }
I20260619 14:43:01.266290 4546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:01.266343 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:01.251818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49482 (local address 127.0.98.62:33549)
0619 14:43:01.252301 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:01.252317 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:01.252353 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:43:01.253100 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:01.253120 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:01.253451 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:01.253814 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:01.254634 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:01.255721 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:01.256606 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:01.257093 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:01.257296 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:01.260266 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:01.260273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:01.260291 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:01.260324 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:01.262411 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:01.262990 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:01.262995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:01.263001 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:01.263201 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:01.263566 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:01.263621 (+ 55us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:01.263622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:01.264443 (+ 821us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:01.264650 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:01.264926 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 14:43:01.265229 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":150,"threads_started":1}
I20260619 14:43:01.268541 4557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:43:01.271756 4557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:43:01.276988 4557 catalog_manager.cc:1347] Generated new cluster ID: d5566b526e2847fab0c5cd52993733d6
I20260619 14:43:01.277060 4557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:43:01.287004 4557 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:43:01.288147 4557 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:43:01.296355 4557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b: Generated new TSK 0
I20260619 14:43:01.297102 4557 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:43:01.331473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33549
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:01.445456 4565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:01.445760 4565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:01.445837 4565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:01.449954 4565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:01.450045 4565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:01.450138 4565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:43:01.455538 4565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:01.456624 4565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:01.457626 4565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:01.467406 4570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:01.467444 4571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:01.467621 4565 server_base.cc:1034] running on GCE node
W20260619 14:43:01.467406 4573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:01.468202 4565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:01.468892 4565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:01.470070 4565 hybrid_clock.cc:648] HybridClock initialized: now 1781880181470055 us; error 25 us; skew 500 ppm
Jun 19 14:43:01 dist-test-slave-cr22 krb5kdc[4459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:01.473001 4565 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:43:01.473765 4565 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:01.473838 4565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:01.474009 4565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:01.476261 4565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "bdc25f31183f421ab81fee8e7acd0221"
format_stamp: "Formatted at 2026-06-19 14:43:01 on dist-test-slave-cr22"
server_key: "9cc855a935e975970c0d73407b39cd6f"
server_key_iv: "1a84733fd2d1be990a919163961d04e8"
server_key_version: "encryptionkey@0"
I20260619 14:43:01.476838 4565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bdc25f31183f421ab81fee8e7acd0221"
format_stamp: "Formatted at 2026-06-19 14:43:01 on dist-test-slave-cr22"
server_key: "9cc855a935e975970c0d73407b39cd6f"
server_key_iv: "1a84733fd2d1be990a919163961d04e8"
server_key_version: "encryptionkey@0"
I20260619 14:43:01.480903 4565 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:43:01.483543 4579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:01.485275 4565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:43:01.485383 4565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "bdc25f31183f421ab81fee8e7acd0221"
format_stamp: "Formatted at 2026-06-19 14:43:01 on dist-test-slave-cr22"
server_key: "9cc855a935e975970c0d73407b39cd6f"
server_key_iv: "1a84733fd2d1be990a919163961d04e8"
server_key_version: "encryptionkey@0"
I20260619 14:43:01.485477 4565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:01.523171 4565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:01.526482 4565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:01.526639 4565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:01.527175 4565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:01.528007 4565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:01.528053 4565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:01.528093 4565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:01.528108 4565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:01.536834 4565 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40185
I20260619 14:43:01.536851 4692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40185 every 8 connection(s)
I20260619 14:43:01.537434 4565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:43:01.538539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 4565
I20260619 14:43:01.538763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:43:01.539119 392 external_mini_cluster.cc:1428] Setting key b6e27f831fc35fbd2627596a5113e745
Jun 19 14:43:01 dist-test-slave-cr22 krb5kdc[4459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:43:01.552387 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:01.539996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33659 (local address 127.0.98.62:33549)
0619 14:43:01.540138 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:01.540143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:01.540572 (+ 429us) server_negotiation.cc:407] Connection header received
0619 14:43:01.542119 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:01.542123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:01.542175 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:01.542300 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:01.543494 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:01.544009 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:01.545161 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:01.545396 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:01.545506 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:01.548123 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:01.548126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:01.548128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:01.548150 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:01.549602 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:01.550295 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:01.550298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:01.550299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:01.550394 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:01.550967 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:01.550970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:01.550972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:01.551228 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:01.551354 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:01.551977 (+ 623us) server_negotiation.cc:299] Negotiation successful
0619 14:43:01.552130 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:43:01.553342 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:01.539850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33549 (local address 127.0.98.1:33659)
0619 14:43:01.540399 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:01.540441 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:01.541910 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:01.542468 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:01.542475 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:01.542842 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:01.543312 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:01.543325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:01.544136 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:01.544140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:01.545031 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:01.545040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:01.545543 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:01.545547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:01.545719 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:01.546379 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:01.546395 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:01.547942 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:01.549755 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:01.549760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:01.549774 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:01.550148 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:01.550533 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:01.550539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:01.550543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:01.550854 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:01.551358 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:01.551364 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:01.551673 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 14:43:01.551902 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 14:43:01.552183 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":128,"threads_started":1}
I20260619 14:43:01.554728 4693 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33549
I20260619 14:43:01.554997 4693 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:01.555609 4693 heartbeater.cc:511] Master 127.0.98.62:33549 requested a full tablet report, sending...
I20260619 14:43:01.557201 4503 ts_manager.cc:194] Registered new tserver with Master: bdc25f31183f421ab81fee8e7acd0221 (127.0.98.1:40185)
I20260619 14:43:01.558377 4503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:43:01.568476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33549
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:01.682111 4698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:01.682365 4698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:01.682415 4698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:01.686544 4698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:01.686615 4698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:01.686676 4698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:43:01.692111 4698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:01.693109 4698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:01.694067 4698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:01.703440 4704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:01.703454 4703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:01.703485 4706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:01.703682 4698 server_base.cc:1034] running on GCE node
I20260619 14:43:01.704177 4698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:01.704819 4698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:01.705996 4698 hybrid_clock.cc:648] HybridClock initialized: now 1781880181705981 us; error 26 us; skew 500 ppm
Jun 19 14:43:01 dist-test-slave-cr22 krb5kdc[4459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:01.708992 4698 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:43:01.709748 4698 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:01.709834 4698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:01.710057 4698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:01.712147 4698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "f27bd4495d284133afce2325da26ad7c"
format_stamp: "Formatted at 2026-06-19 14:43:01 on dist-test-slave-cr22"
server_key: "552aa0f45bc63a52d7a42edd761cf2a5"
server_key_iv: "8ac11042e6f3f54321fde148dbb8f3c9"
server_key_version: "encryptionkey@0"
I20260619 14:43:01.712705 4698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f27bd4495d284133afce2325da26ad7c"
format_stamp: "Formatted at 2026-06-19 14:43:01 on dist-test-slave-cr22"
server_key: "552aa0f45bc63a52d7a42edd761cf2a5"
server_key_iv: "8ac11042e6f3f54321fde148dbb8f3c9"
server_key_version: "encryptionkey@0"
I20260619 14:43:01.716521 4698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:43:01.719218 4712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:01.720636 4698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:43:01.720762 4698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "f27bd4495d284133afce2325da26ad7c"
format_stamp: "Formatted at 2026-06-19 14:43:01 on dist-test-slave-cr22"
server_key: "552aa0f45bc63a52d7a42edd761cf2a5"
server_key_iv: "8ac11042e6f3f54321fde148dbb8f3c9"
server_key_version: "encryptionkey@0"
I20260619 14:43:01.720934 4698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:01.741114 4698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:01.744463 4698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:01.744690 4698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:01.745434 4698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:01.746395 4698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:01.746448 4698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:01.746490 4698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:01.746546 4698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:01.755774 4698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35693
I20260619 14:43:01.755802 4825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35693 every 8 connection(s)
I20260619 14:43:01.756392 4698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:43:01.765679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 4698
I20260619 14:43:01.765816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:43:01.766135 392 external_mini_cluster.cc:1428] Setting key 7f008ade71ec1078fd8e04f75c36d88f
Jun 19 14:43:01 dist-test-slave-cr22 krb5kdc[4459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:43:01.771507 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:01.758342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55165 (local address 127.0.98.62:33549)
0619 14:43:01.758518 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:01.758522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:01.759518 (+ 996us) server_negotiation.cc:407] Connection header received
0619 14:43:01.760898 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:01.760902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:01.760956 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:01.761039 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:01.762216 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:01.762762 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:01.764000 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:01.764231 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:01.764350 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:01.767245 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:01.767249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:01.767251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:01.767274 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:01.768818 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:01.769553 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:01.769556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:01.769557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:01.769735 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:01.770243 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:01.770246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:01.770247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:01.770433 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:01.770514 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:01.771211 (+ 697us) server_negotiation.cc:299] Negotiation successful
0619 14:43:01.771318 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 14:43:01.772789 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:01.758889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33549 (local address 127.0.98.2:55165)
0619 14:43:01.759340 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:01.759380 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:01.760680 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:01.761215 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:01.761222 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:01.761601 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:01.762074 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:01.762087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:01.762889 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:01.762892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:01.763850 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:01.763860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:01.764386 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:01.764390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:01.764560 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:01.765271 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:01.765301 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:01.767076 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:01.768968 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:01.768975 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:01.768991 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:01.769403 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:01.769840 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:01.769846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:01.769855 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:01.770061 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:01.770543 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:01.770553 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:01.770904 (+ 351us) client_negotiation.cc:769] Sending connection context
0619 14:43:01.771140 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 14:43:01.771576 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":164,"threads_started":1}
I20260619 14:43:01.774353 4826 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33549
I20260619 14:43:01.774611 4826 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:01.775127 4826 heartbeater.cc:511] Master 127.0.98.62:33549 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:43:01.776353 4503 ts_manager.cc:194] Registered new tserver with Master: f27bd4495d284133afce2325da26ad7c (127.0.98.2:35693)
I20260619 14:43:01.776954 4503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:43:01.795420 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33549
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:01.905464 4831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:01.905731 4831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:01.905803 4831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:01.909818 4831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:01.909904 4831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:01.909988 4831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:43:01.915117 4831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:01.916200 4831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:01.917186 4831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:01.926936 4839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:01.926950 4837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:01.927062 4831 server_base.cc:1034] running on GCE node
W20260619 14:43:01.927290 4836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:01.927767 4831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:01.928498 4831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:01.929669 4831 hybrid_clock.cc:648] HybridClock initialized: now 1781880181929627 us; error 55 us; skew 500 ppm
Jun 19 14:43:01 dist-test-slave-cr22 krb5kdc[4459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:01.932761 4831 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:43:01.933521 4831 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:01.933584 4831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:01.933832 4831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:01.936015 4831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "649b0bfbb7e74458a307f33956fa0621"
format_stamp: "Formatted at 2026-06-19 14:43:01 on dist-test-slave-cr22"
server_key: "b2e24f5214b61b1200bab5d0e6e548e9"
server_key_iv: "f98826867a3c07c34442c9cea5603a29"
server_key_version: "encryptionkey@0"
I20260619 14:43:01.936556 4831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "649b0bfbb7e74458a307f33956fa0621"
format_stamp: "Formatted at 2026-06-19 14:43:01 on dist-test-slave-cr22"
server_key: "b2e24f5214b61b1200bab5d0e6e548e9"
server_key_iv: "f98826867a3c07c34442c9cea5603a29"
server_key_version: "encryptionkey@0"
I20260619 14:43:01.940444 4831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:43:01.942991 4845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:01.944291 4831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:43:01.944414 4831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "649b0bfbb7e74458a307f33956fa0621"
format_stamp: "Formatted at 2026-06-19 14:43:01 on dist-test-slave-cr22"
server_key: "b2e24f5214b61b1200bab5d0e6e548e9"
server_key_iv: "f98826867a3c07c34442c9cea5603a29"
server_key_version: "encryptionkey@0"
I20260619 14:43:01.944525 4831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:01.969163 4831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:01.972487 4831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:01.972697 4831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:01.973279 4831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:01.974112 4831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:01.974159 4831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:01.974224 4831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:01.974257 4831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:01.983384 4831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37565
I20260619 14:43:01.983448 4958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37565 every 8 connection(s)
I20260619 14:43:01.984014 4831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:43:01.992202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 4831
I20260619 14:43:01.992369 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:43:01.992717 392 external_mini_cluster.cc:1428] Setting key 98c865783e9c31382a909ffacccf62c3
Jun 19 14:43:01 dist-test-slave-cr22 krb5kdc[4459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:43:01.998378 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:01.985739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57383 (local address 127.0.98.62:33549)
0619 14:43:01.986086 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:01.986096 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:01.986634 (+ 538us) server_negotiation.cc:407] Connection header received
0619 14:43:01.988098 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:01.988102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:01.988162 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:01.988266 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:01.989566 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:01.990106 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:01.991292 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:01.991594 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:01.991700 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:01.994714 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:01.994718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:01.994721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:01.994749 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:01.996287 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:01.996838 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:01.996845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:01.996849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:01.996936 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:01.997267 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:01.997271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:01.997273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:01.997466 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:01.997570 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:01.998088 (+ 518us) server_negotiation.cc:299] Negotiation successful
0619 14:43:01.998206 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
I20260619 14:43:01.999210 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:01.986057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33549 (local address 127.0.98.3:57383)
0619 14:43:01.986495 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:01.986537 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:01.987876 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:01.988444 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:01.988451 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:01.988854 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:01.989386 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:01.989399 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:01.990244 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:01.990248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:01.991152 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:01.991162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:01.991757 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:01.991761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:01.991998 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:01.992756 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:01.992778 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:01.994549 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:01.996436 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:01.996442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:01.996459 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:01.996732 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:01.997025 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:01.997029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:01.997031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:01.997174 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:01.997575 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:01.997584 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:01.997856 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 14:43:01.998049 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 14:43:01.998380 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
I20260619 14:43:02.000275 4959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33549
I20260619 14:43:02.000495 4959 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:02.000903 4959 heartbeater.cc:511] Master 127.0.98.62:33549 requested a full tablet report, sending...
I20260619 14:43:02.001824 4503 ts_manager.cc:194] Registered new tserver with Master: 649b0bfbb7e74458a307f33956fa0621 (127.0.98.3:37565)
I20260619 14:43:02.002425 4503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57383
I20260619 14:43:02.006623 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:43:02.016323 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:02.008605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.0.98.62:33549)
0619 14:43:02.008716 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:02.008720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:02.008973 (+ 253us) server_negotiation.cc:407] Connection header received
0619 14:43:02.009017 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:02.009019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:02.009059 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:02.009132 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:02.009828 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.010334 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.011211 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.011420 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.011580 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:02.012891 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:02.012893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:02.012895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:02.012917 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:02.014368 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:02.014877 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:02.014884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:02.014887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:02.014994 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:02.015263 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:02.015266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:02.015267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:02.015687 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:02.015794 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:02.016092 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 14:43:02.016194 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:43:02.019351 4503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:43:02.022398 4503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:43:02.037175 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:02.031630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37565 (local address 127.0.0.1:52736)
0619 14:43:02.032306 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:02.032319 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:02.032402 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:02.033060 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:02.033063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:02.033081 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:02.033159 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:02.033164 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.034620 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.034623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:02.035573 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:02.035583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.036607 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.036611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:02.036728 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:02.036766 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 14:43:02.036890 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 14:43:02.037015 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":33,"thread_start_us":105,"threads_started":1}
I20260619 14:43:02.037230 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:02.031816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.0.98.3:37565)
0619 14:43:02.032660 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:02.032671 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:02.032692 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:43:02.032762 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:02.032767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:02.032941 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:02.033088 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:02.033340 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.034469 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.035696 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.036457 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.036581 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:02.036761 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:02.036985 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 14:43:02.037084 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":98,"threads_started":1}
I20260619 14:43:02.037355 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:02.030852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40185 (local address 127.0.0.1:47200)
0619 14:43:02.031422 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:02.031514 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:02.031657 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:02.032511 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:02.032518 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:02.032540 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:02.032741 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:02.032749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.034591 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.034595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:02.035801 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:02.035811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.036868 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.036871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:02.036994 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:02.037018 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:43:02.037115 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 14:43:02.037188 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":59,"thread_start_us":87,"threads_started":1}
I20260619 14:43:02.037638 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:02.031709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47200 (local address 127.0.98.1:40185)
0619 14:43:02.031990 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:02.031997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:02.032022 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:43:02.032105 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:02.032111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:02.032333 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:02.032894 (+ 561us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:02.033066 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.034442 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.035929 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.036703 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.036849 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:02.036978 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:02.037145 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 14:43:02.037241 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":84,"threads_started":1}
I20260619 14:43:02.037899 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:02.030878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35693 (local address 127.0.0.1:47210)
0619 14:43:02.031916 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:02.031932 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:02.032033 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:02.032734 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:02.032737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:02.032755 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:02.032860 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:02.032866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.033967 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.033973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:02.035860 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:02.035869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.036864 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.036869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:02.037451 (+ 582us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:02.037555 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 14:43:02.037696 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 14:43:02.037780 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"mutex_wait_us":339,"thread_start_us":81,"threads_started":1}
I20260619 14:43:02.038280 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:02.031094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47210 (local address 127.0.98.2:35693)
0619 14:43:02.032288 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:02.032304 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:02.032336 (+ 32us) server_negotiation.cc:407] Connection header received
0619 14:43:02.032411 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:02.032418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:02.032608 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:02.032762 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:02.032974 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.033841 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.035968 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.036512 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.036657 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:02.036800 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:02.037748 (+ 948us) server_negotiation.cc:299] Negotiation successful
0619 14:43:02.037870 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1078,"thread_start_us":114,"threads_started":1}
I20260619 14:43:02.040596 4893 tablet_service.cc:1467] Processing CreateTablet for tablet 26929483f58a46cb918dff6d72cd832d (DEFAULT_TABLE table=test-table [id=ab45e100891640f78d1c802286d78b3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:02.040743 4627 tablet_service.cc:1467] Processing CreateTablet for tablet 26929483f58a46cb918dff6d72cd832d (DEFAULT_TABLE table=test-table [id=ab45e100891640f78d1c802286d78b3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:02.040743 4760 tablet_service.cc:1467] Processing CreateTablet for tablet 26929483f58a46cb918dff6d72cd832d (DEFAULT_TABLE table=test-table [id=ab45e100891640f78d1c802286d78b3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:02.041646 4893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26929483f58a46cb918dff6d72cd832d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:02.041754 4760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26929483f58a46cb918dff6d72cd832d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:02.041754 4627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26929483f58a46cb918dff6d72cd832d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:02.048426 4976 tablet_bootstrap.cc:492] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621: Bootstrap starting.
I20260619 14:43:02.049369 4977 tablet_bootstrap.cc:492] T 26929483f58a46cb918dff6d72cd832d P bdc25f31183f421ab81fee8e7acd0221: Bootstrap starting.
I20260619 14:43:02.051065 4977 tablet_bootstrap.cc:654] T 26929483f58a46cb918dff6d72cd832d P bdc25f31183f421ab81fee8e7acd0221: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:02.051097 4976 tablet_bootstrap.cc:654] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:02.051824 4978 tablet_bootstrap.cc:492] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c: Bootstrap starting.
I20260619 14:43:02.052484 4977 log.cc:824] T 26929483f58a46cb918dff6d72cd832d P bdc25f31183f421ab81fee8e7acd0221: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:02.052524 4976 log.cc:824] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:02.054318 4978 tablet_bootstrap.cc:654] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:02.054895 4976 tablet_bootstrap.cc:492] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621: No bootstrap required, opened a new log
I20260619 14:43:02.055117 4976 ts_tablet_manager.cc:1397] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260619 14:43:02.055509 4977 tablet_bootstrap.cc:492] T 26929483f58a46cb918dff6d72cd832d P bdc25f31183f421ab81fee8e7acd0221: No bootstrap required, opened a new log
I20260619 14:43:02.055624 4978 log.cc:824] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:02.055724 4977 ts_tablet_manager.cc:1397] T 26929483f58a46cb918dff6d72cd832d P bdc25f31183f421ab81fee8e7acd0221: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260619 14:43:02.057673 4978 tablet_bootstrap.cc:492] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c: No bootstrap required, opened a new log
I20260619 14:43:02.057873 4978 ts_tablet_manager.cc:1397] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:43:02.058919 4977 raft_consensus.cc:348] T 26929483f58a46cb918dff6d72cd832d P bdc25f31183f421ab81fee8e7acd0221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc25f31183f421ab81fee8e7acd0221" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40185 } } peers { permanent_uuid: "f27bd4495d284133afce2325da26ad7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35693 } } peers { permanent_uuid: "649b0bfbb7e74458a307f33956fa0621" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37565 } }
I20260619 14:43:02.059160 4977 raft_consensus.cc:374] T 26929483f58a46cb918dff6d72cd832d P bdc25f31183f421ab81fee8e7acd0221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:02.059034 4976 raft_consensus.cc:348] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc25f31183f421ab81fee8e7acd0221" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40185 } } peers { permanent_uuid: "f27bd4495d284133afce2325da26ad7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35693 } } peers { permanent_uuid: "649b0bfbb7e74458a307f33956fa0621" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37565 } }
I20260619 14:43:02.059240 4977 raft_consensus.cc:729] T 26929483f58a46cb918dff6d72cd832d P bdc25f31183f421ab81fee8e7acd0221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdc25f31183f421ab81fee8e7acd0221, State: Initialized, Role: FOLLOWER
I20260619 14:43:02.059305 4976 raft_consensus.cc:374] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:02.059366 4976 raft_consensus.cc:729] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649b0bfbb7e74458a307f33956fa0621, State: Initialized, Role: FOLLOWER
I20260619 14:43:02.059823 4977 consensus_queue.cc:260] T 26929483f58a46cb918dff6d72cd832d P bdc25f31183f421ab81fee8e7acd0221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc25f31183f421ab81fee8e7acd0221" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40185 } } peers { permanent_uuid: "f27bd4495d284133afce2325da26ad7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35693 } } peers { permanent_uuid: "649b0bfbb7e74458a307f33956fa0621" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37565 } }
I20260619 14:43:02.059820 4976 consensus_queue.cc:260] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc25f31183f421ab81fee8e7acd0221" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40185 } } peers { permanent_uuid: "f27bd4495d284133afce2325da26ad7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35693 } } peers { permanent_uuid: "649b0bfbb7e74458a307f33956fa0621" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37565 } }
I20260619 14:43:02.060441 4693 heartbeater.cc:503] Master 127.0.98.62:33549 was elected leader, sending a full tablet report...
I20260619 14:43:02.060804 4977 ts_tablet_manager.cc:1428] T 26929483f58a46cb918dff6d72cd832d P bdc25f31183f421ab81fee8e7acd0221: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20260619 14:43:02.060806 4976 ts_tablet_manager.cc:1428] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:43:02.061117 4959 heartbeater.cc:503] Master 127.0.98.62:33549 was elected leader, sending a full tablet report...
I20260619 14:43:02.061666 4978 raft_consensus.cc:348] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc25f31183f421ab81fee8e7acd0221" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40185 } } peers { permanent_uuid: "f27bd4495d284133afce2325da26ad7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35693 } } peers { permanent_uuid: "649b0bfbb7e74458a307f33956fa0621" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37565 } }
I20260619 14:43:02.061946 4978 raft_consensus.cc:374] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:02.062041 4978 raft_consensus.cc:729] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f27bd4495d284133afce2325da26ad7c, State: Initialized, Role: FOLLOWER
I20260619 14:43:02.062481 4978 consensus_queue.cc:260] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc25f31183f421ab81fee8e7acd0221" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40185 } } peers { permanent_uuid: "f27bd4495d284133afce2325da26ad7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35693 } } peers { permanent_uuid: "649b0bfbb7e74458a307f33956fa0621" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37565 } }
I20260619 14:43:02.063109 4826 heartbeater.cc:503] Master 127.0.98.62:33549 was elected leader, sending a full tablet report...
I20260619 14:43:02.063416 4978 ts_tablet_manager.cc:1428] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:43:02.201529 4984 raft_consensus.cc:482] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:43:02.201867 4984 raft_consensus.cc:504] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc25f31183f421ab81fee8e7acd0221" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40185 } } peers { permanent_uuid: "f27bd4495d284133afce2325da26ad7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35693 } } peers { permanent_uuid: "649b0bfbb7e74458a307f33956fa0621" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37565 } }
I20260619 14:43:02.203035 4984 leader_election.cc:288] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdc25f31183f421ab81fee8e7acd0221 (127.0.98.1:40185), 649b0bfbb7e74458a307f33956fa0621 (127.0.98.3:37565)
I20260619 14:43:02.208163 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:02.203926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59373 (local address 127.0.98.3:37565)
0619 14:43:02.204057 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:02.204062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:02.204144 (+ 82us) server_negotiation.cc:407] Connection header received
0619 14:43:02.204238 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:02.204240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:02.204302 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:02.204389 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:02.204612 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.205353 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.206728 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.207340 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.207478 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:02.207555 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:02.207964 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 14:43:02.208033 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:43:02.208163 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:02.203560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50343 (local address 127.0.98.1:40185)
0619 14:43:02.203696 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:02.203701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:02.203719 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:43:02.203781 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:02.203784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:02.203836 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:02.203936 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:02.204316 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.205165 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.206728 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.207347 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.207478 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:02.207555 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:02.207964 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 14:43:02.208034 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:43:02.208248 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:02.203359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40185 (local address 127.0.98.2:50343)
0619 14:43:02.203544 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:02.203562 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:02.203666 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:02.204033 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:02.204036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:02.204054 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:02.204200 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:02.204210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.205305 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.205308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:02.206588 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:02.206599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.207482 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.207486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:02.207740 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:02.207854 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 14:43:02.207935 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:43:02.208085 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":149}
I20260619 14:43:02.208256 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:02.203713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37565 (local address 127.0.98.2:59373)
0619 14:43:02.204055 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:02.204068 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:02.204159 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:02.204406 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:02.204409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:02.204423 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:02.204511 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:02.204515 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.205451 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.205454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:02.206618 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:02.206626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.207482 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.207486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:02.207742 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:02.207853 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 14:43:02.207935 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 14:43:02.208085 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":169,"thread_start_us":156,"threads_started":1}
I20260619 14:43:02.208956 4913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26929483f58a46cb918dff6d72cd832d" candidate_uuid: "f27bd4495d284133afce2325da26ad7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649b0bfbb7e74458a307f33956fa0621" is_pre_election: true
I20260619 14:43:02.208957 4647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26929483f58a46cb918dff6d72cd832d" candidate_uuid: "f27bd4495d284133afce2325da26ad7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdc25f31183f421ab81fee8e7acd0221" is_pre_election: true
I20260619 14:43:02.209251 4913 raft_consensus.cc:2454] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f27bd4495d284133afce2325da26ad7c in term 0.
I20260619 14:43:02.209259 4647 raft_consensus.cc:2454] T 26929483f58a46cb918dff6d72cd832d P bdc25f31183f421ab81fee8e7acd0221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f27bd4495d284133afce2325da26ad7c in term 0.
I20260619 14:43:02.209738 4716 leader_election.cc:302] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649b0bfbb7e74458a307f33956fa0621, f27bd4495d284133afce2325da26ad7c; no voters:
I20260619 14:43:02.210014 4984 raft_consensus.cc:2781] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:43:02.210093 4984 raft_consensus.cc:482] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:43:02.210124 4984 raft_consensus.cc:3037] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:02.211165 4984 raft_consensus.cc:504] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc25f31183f421ab81fee8e7acd0221" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40185 } } peers { permanent_uuid: "f27bd4495d284133afce2325da26ad7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35693 } } peers { permanent_uuid: "649b0bfbb7e74458a307f33956fa0621" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37565 } }
I20260619 14:43:02.211557 4984 leader_election.cc:288] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [CANDIDATE]: Term 1 election: Requested vote from peers bdc25f31183f421ab81fee8e7acd0221 (127.0.98.1:40185), 649b0bfbb7e74458a307f33956fa0621 (127.0.98.3:37565)
I20260619 14:43:02.211889 4647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26929483f58a46cb918dff6d72cd832d" candidate_uuid: "f27bd4495d284133afce2325da26ad7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdc25f31183f421ab81fee8e7acd0221"
I20260619 14:43:02.211973 4913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26929483f58a46cb918dff6d72cd832d" candidate_uuid: "f27bd4495d284133afce2325da26ad7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649b0bfbb7e74458a307f33956fa0621"
I20260619 14:43:02.212033 4647 raft_consensus.cc:3037] T 26929483f58a46cb918dff6d72cd832d P bdc25f31183f421ab81fee8e7acd0221 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:02.212122 4913 raft_consensus.cc:3037] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:02.213078 4647 raft_consensus.cc:2454] T 26929483f58a46cb918dff6d72cd832d P bdc25f31183f421ab81fee8e7acd0221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f27bd4495d284133afce2325da26ad7c in term 1.
I20260619 14:43:02.213286 4913 raft_consensus.cc:2454] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f27bd4495d284133afce2325da26ad7c in term 1.
I20260619 14:43:02.213399 4716 leader_election.cc:302] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdc25f31183f421ab81fee8e7acd0221, f27bd4495d284133afce2325da26ad7c; no voters:
I20260619 14:43:02.213567 4984 raft_consensus.cc:2781] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:02.213843 4984 raft_consensus.cc:686] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [term 1 LEADER]: Becoming Leader. State: Replica: f27bd4495d284133afce2325da26ad7c, State: Running, Role: LEADER
I20260619 14:43:02.214187 4984 consensus_queue.cc:237] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc25f31183f421ab81fee8e7acd0221" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40185 } } peers { permanent_uuid: "f27bd4495d284133afce2325da26ad7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35693 } } peers { permanent_uuid: "649b0bfbb7e74458a307f33956fa0621" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37565 } }
I20260619 14:43:02.217379 4503 catalog_manager.cc:5495] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c reported cstate change: term changed from 0 to 1, leader changed from <none> to f27bd4495d284133afce2325da26ad7c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f27bd4495d284133afce2325da26ad7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdc25f31183f421ab81fee8e7acd0221" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f27bd4495d284133afce2325da26ad7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "649b0bfbb7e74458a307f33956fa0621" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37565 } health_report { overall_health: UNKNOWN } } }
I20260619 14:43:02.232451 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:02.228447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47222 (local address 127.0.98.2:35693)
0619 14:43:02.228570 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:02.228573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:02.228587 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:43:02.228826 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:02.228830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:02.228899 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:02.229009 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:43:02.229286 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.229910 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.230830 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:02.231038 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:02.231124 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:02.231422 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:43:02.232051 (+ 629us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:43:02.232151 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:02.232284 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 14:43:02.232334 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 14:43:02.234911 4960 tablet.cc:2369] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:43:02.238309 4913 raft_consensus.cc:1261] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621 [term 1 FOLLOWER]: Refusing update from remote peer f27bd4495d284133afce2325da26ad7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:02.238320 4647 raft_consensus.cc:1261] T 26929483f58a46cb918dff6d72cd832d P bdc25f31183f421ab81fee8e7acd0221 [term 1 FOLLOWER]: Refusing update from remote peer f27bd4495d284133afce2325da26ad7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:02.239073 4984 consensus_queue.cc:1035] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdc25f31183f421ab81fee8e7acd0221" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:02.239315 4986 consensus_queue.cc:1035] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b0bfbb7e74458a307f33956fa0621" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:02.257418 4503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:43:02.257617 4503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:43:02.260396 4503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cf0ee81227344c6986199dbd922970b: Sending DeleteTablet for 3 replicas of tablet 26929483f58a46cb918dff6d72cd832d
I20260619 14:43:02.261188 4760 tablet_service.cc:1514] Processing DeleteTablet for tablet 26929483f58a46cb918dff6d72cd832d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47210
I20260619 14:43:02.261293 4893 tablet_service.cc:1514] Processing DeleteTablet for tablet 26929483f58a46cb918dff6d72cd832d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52736
I20260619 14:43:02.261399 4627 tablet_service.cc:1514] Processing DeleteTablet for tablet 26929483f58a46cb918dff6d72cd832d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47200
I20260619 14:43:02.261720 4998 tablet_replica.cc:330] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c: stopping tablet replica
I20260619 14:43:02.262030 4998 raft_consensus.cc:2229] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:43:02.262426 4998 raft_consensus.cc:2258] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:02.263279 4991 mvcc.cc:205] Tried to move back new op lower bound from 7298581226440679424 to 7298581226352025600. Current Snapshot: MvccSnapshot[applied={T|T < 7298581226440679424}]
I20260619 14:43:02.263545 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 4565
I20260619 14:43:02.264303 4998 ts_tablet_manager.cc:1905] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:02.264319 4999 tablet_replica.cc:330] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621: stopping tablet replica
I20260619 14:43:02.264724 4999 raft_consensus.cc:2229] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:43:02.264986 4999 raft_consensus.cc:2258] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:02.267321 4998 ts_tablet_manager.cc:1918] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:43:02.267403 4998 log.cc:1196] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/26929483f58a46cb918dff6d72cd832d
I20260619 14:43:02.267876 4998 ts_tablet_manager.cc:1939] T 26929483f58a46cb918dff6d72cd832d P f27bd4495d284133afce2325da26ad7c: Deleting consensus metadata
I20260619 14:43:02.268728 4999 ts_tablet_manager.cc:1905] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:02.269053 4488 catalog_manager.cc:4841] TS f27bd4495d284133afce2325da26ad7c (127.0.98.2:35693): tablet 26929483f58a46cb918dff6d72cd832d (table test-table [id=ab45e100891640f78d1c802286d78b3e]) successfully deleted
I20260619 14:43:02.271013 4999 ts_tablet_manager.cc:1918] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:43:02.271133 4999 log.cc:1196] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/26929483f58a46cb918dff6d72cd832d
I20260619 14:43:02.271522 4999 ts_tablet_manager.cc:1939] T 26929483f58a46cb918dff6d72cd832d P 649b0bfbb7e74458a307f33956fa0621: Deleting consensus metadata
I20260619 14:43:02.272703 4490 catalog_manager.cc:4841] TS 649b0bfbb7e74458a307f33956fa0621 (127.0.98.3:37565): tablet 26929483f58a46cb918dff6d72cd832d (table test-table [id=ab45e100891640f78d1c802286d78b3e]) successfully deleted
I20260619 14:43:02.276789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 4698
W20260619 14:43:02.276793 4490 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40185 (error 108)
I20260619 14:43:02.278687 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:02.278405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40185 (local address 127.0.0.1:47202)
0619 14:43:02.278544 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:02.278615 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 14:43:02.278874 4490 catalog_manager.cc:4584] TS bdc25f31183f421ab81fee8e7acd0221 (127.0.98.1:40185): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 26929483f58a46cb918dff6d72cd832d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40185: connect: Connection refused (error 111)
I20260619 14:43:02.283416 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 4831
I20260619 14:43:02.288969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 4472
2026-06-19T14:43:02Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2455 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T14:43:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:43:02Z Disabled control of system clock
I20260619 14:43:02.318984 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34047
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 14:43:02.427423 5006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:02.427713 5006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:02.427752 5006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:02.431767 5006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:43:02.431828 5006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:02.431844 5006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:02.431861 5006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:43:02.431876 5006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:43:02.436960 5006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34047
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:02.438059 5006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:02.439420 5006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:02.448177 5011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:02.448222 5006 server_base.cc:1034] running on GCE node
W20260619 14:43:02.448294 5012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:02.448307 5014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:02.448966 5006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:02.450035 5006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:02.451218 5006 hybrid_clock.cc:648] HybridClock initialized: now 1781880182451200 us; error 31 us; skew 500 ppm
I20260619 14:43:02.453166 5006 webserver.cc:460] Webserver started at http://127.0.98.62:40101/ using document root <none> and password file <none>
I20260619 14:43:02.453773 5006 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:02.453868 5006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:02.454097 5006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:02.456477 5006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "4b52b524adde46d1ba1c8966e81d6f60"
format_stamp: "Formatted at 2026-06-19 14:43:02 on dist-test-slave-cr22"
server_key: "6683b64de81c8d420191db1e98c6759e"
server_key_iv: "b48b945a35944daa2331487872e1c381"
server_key_version: "encryptionkey@0"
I20260619 14:43:02.457141 5006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "4b52b524adde46d1ba1c8966e81d6f60"
format_stamp: "Formatted at 2026-06-19 14:43:02 on dist-test-slave-cr22"
server_key: "6683b64de81c8d420191db1e98c6759e"
server_key_iv: "b48b945a35944daa2331487872e1c381"
server_key_version: "encryptionkey@0"
I20260619 14:43:02.460975 5006 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:43:02.463610 5020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:02.465051 5006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:43:02.465188 5006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "4b52b524adde46d1ba1c8966e81d6f60"
format_stamp: "Formatted at 2026-06-19 14:43:02 on dist-test-slave-cr22"
server_key: "6683b64de81c8d420191db1e98c6759e"
server_key_iv: "b48b945a35944daa2331487872e1c381"
server_key_version: "encryptionkey@0"
I20260619 14:43:02.465302 5006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:02.469894 5006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:02.470484 5006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:02.470664 5006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:02.477555 5006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34047
I20260619 14:43:02.477577 5072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34047 every 8 connection(s)
I20260619 14:43:02.478497 5006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:43:02.481731 5073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:02.484575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 5006
I20260619 14:43:02.484719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:43:02.484999 392 external_mini_cluster.cc:1428] Setting key 4ca99c67c236a7682bbbf134b2ec5fb4
I20260619 14:43:02.487989 5073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60: Bootstrap starting.
I20260619 14:43:02.490751 5073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:02.491422 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:02.486485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.0.98.62:34047)
0619 14:43:02.487203 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:02.487224 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:02.487267 (+ 43us) server_negotiation.cc:407] Connection header received
0619 14:43:02.488123 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:02.488145 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:02.488580 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:02.488948 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:02.489733 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:02.489738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:02.489753 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:02.489764 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:02.489838 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:02.489860 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:02.489991 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:02.490216 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 14:43:02.490462 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":127,"threads_started":1}
I20260619 14:43:02.491706 5073 log.cc:824] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:02.494408 5073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60: No bootstrap required, opened a new log
I20260619 14:43:02.498018 5073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b52b524adde46d1ba1c8966e81d6f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34047 } }
I20260619 14:43:02.498340 5073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:02.498427 5073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b52b524adde46d1ba1c8966e81d6f60, State: Initialized, Role: FOLLOWER
I20260619 14:43:02.498893 5073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b52b524adde46d1ba1c8966e81d6f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34047 } }
I20260619 14:43:02.499037 5073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:43:02.499102 5073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:43:02.499209 5073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:02.500334 5073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b52b524adde46d1ba1c8966e81d6f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34047 } }
I20260619 14:43:02.500877 5073 leader_election.cc:302] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b52b524adde46d1ba1c8966e81d6f60; no voters:
I20260619 14:43:02.501221 5073 leader_election.cc:288] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:43:02.501372 5078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:02.501619 5078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60 [term 1 LEADER]: Becoming Leader. State: Replica: 4b52b524adde46d1ba1c8966e81d6f60, State: Running, Role: LEADER
I20260619 14:43:02.502009 5078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b52b524adde46d1ba1c8966e81d6f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34047 } }
I20260619 14:43:02.502576 5073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:43:02.503816 5079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b52b524adde46d1ba1c8966e81d6f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b52b524adde46d1ba1c8966e81d6f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34047 } } }
I20260619 14:43:02.503940 5079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:02.503907 5080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b52b524adde46d1ba1c8966e81d6f60. Latest consensus state: current_term: 1 leader_uuid: "4b52b524adde46d1ba1c8966e81d6f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b52b524adde46d1ba1c8966e81d6f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34047 } } }
I20260619 14:43:02.504009 5080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:02.504402 5087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:43:02.507591 5087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:43:02.513304 5087 catalog_manager.cc:1347] Generated new cluster ID: 9c9c5399dbdd42f6b8a7fec1956c0765
I20260619 14:43:02.513399 5087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:43:02.543538 5087 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:43:02.543957 5087 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:43:02.552506 5087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60: Generated new TSK 0
I20260619 14:43:02.553328 5087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:43:02.560779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34047
--builtin_ntp_servers=127.0.98.20:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 14:43:02.674407 5097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:02.674683 5097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:02.674724 5097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:02.678761 5097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:02.678819 5097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:02.678877 5097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:43:02.683607 5097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:02.684650 5097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:02.685971 5097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:02.695843 5102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:02.695956 5103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:02.695979 5105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:02.696131 5097 server_base.cc:1034] running on GCE node
I20260619 14:43:02.696579 5097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:02.697348 5097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:02.698580 5097 hybrid_clock.cc:648] HybridClock initialized: now 1781880182698560 us; error 32 us; skew 500 ppm
I20260619 14:43:02.700596 5097 webserver.cc:460] Webserver started at http://127.0.98.1:44007/ using document root <none> and password file <none>
I20260619 14:43:02.701205 5097 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:02.701295 5097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:02.701526 5097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:02.703794 5097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "8a82737d751b4ccd8a0c5ac180625603"
format_stamp: "Formatted at 2026-06-19 14:43:02 on dist-test-slave-cr22"
server_key: "318b26593ca103a58c39d8eed99d7d36"
server_key_iv: "c9fa46c5074bdddbf2f8d420720ae1db"
server_key_version: "encryptionkey@0"
I20260619 14:43:02.704408 5097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8a82737d751b4ccd8a0c5ac180625603"
format_stamp: "Formatted at 2026-06-19 14:43:02 on dist-test-slave-cr22"
server_key: "318b26593ca103a58c39d8eed99d7d36"
server_key_iv: "c9fa46c5074bdddbf2f8d420720ae1db"
server_key_version: "encryptionkey@0"
I20260619 14:43:02.708250 5097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 14:43:02.710832 5111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:02.712208 5097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:43:02.712348 5097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "8a82737d751b4ccd8a0c5ac180625603"
format_stamp: "Formatted at 2026-06-19 14:43:02 on dist-test-slave-cr22"
server_key: "318b26593ca103a58c39d8eed99d7d36"
server_key_iv: "c9fa46c5074bdddbf2f8d420720ae1db"
server_key_version: "encryptionkey@0"
I20260619 14:43:02.712471 5097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:02.717186 5097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:02.717823 5097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:02.718001 5097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:02.718559 5097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:02.719568 5097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:02.719615 5097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:02.719674 5097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:02.719704 5097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:02.728253 5097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39919
I20260619 14:43:02.728341 5224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39919 every 8 connection(s)
I20260619 14:43:02.729234 5097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:43:02.735822 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:02.731397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42271 (local address 127.0.98.62:34047)
0619 14:43:02.731579 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:02.731583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:02.732651 (+ 1068us) server_negotiation.cc:407] Connection header received
0619 14:43:02.734034 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:02.734038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:02.734083 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:02.734168 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:02.735310 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:02.735314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:02.735317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:02.735335 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:02.735355 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:02.735363 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:02.735502 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:02.735634 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 14:43:02.735681 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 14:43:02.736692 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:02.731799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34047 (local address 127.0.98.1:42271)
0619 14:43:02.732502 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:02.732544 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:02.733827 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:02.734328 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:02.734334 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:02.734424 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:02.735023 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:43:02.735024 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:02.735064 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:43:02.735134 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:02.735485 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:02.735494 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:02.735499 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:43:02.735680 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 14:43:02.735937 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":111,"threads_started":1}
I20260619 14:43:02.737931 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 5097
I20260619 14:43:02.738067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:43:02.738162 5225 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34047
I20260619 14:43:02.738397 392 external_mini_cluster.cc:1428] Setting key 1ba10c73168b298fa613f2c4f3b7571c
I20260619 14:43:02.738430 5225 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:02.738829 5225 heartbeater.cc:511] Master 127.0.98.62:34047 requested a full tablet report, sending...
I20260619 14:43:02.740378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34047
--builtin_ntp_servers=127.0.98.20:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 14:43:02.740525 5037 ts_manager.cc:194] Registered new tserver with Master: 8a82737d751b4ccd8a0c5ac180625603 (127.0.98.1:39919)
W20260619 14:43:02.854888 5228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:02.855183 5228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:02.855283 5228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:02.859503 5228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:02.859583 5228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:02.859683 5228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:43:02.864495 5228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:02.865607 5228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:02.866956 5228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:02.876961 5236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:02.877029 5234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:02.877023 5228 server_base.cc:1034] running on GCE node
W20260619 14:43:02.877064 5233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:02.877692 5228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:02.878343 5228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:02.879546 5228 hybrid_clock.cc:648] HybridClock initialized: now 1781880182879491 us; error 67 us; skew 500 ppm
I20260619 14:43:02.881395 5228 webserver.cc:460] Webserver started at http://127.0.98.2:40837/ using document root <none> and password file <none>
I20260619 14:43:02.881968 5228 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:02.882095 5228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:02.882321 5228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:02.884460 5228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "87626762ccf9441d8bf7cc58f0d98f28"
format_stamp: "Formatted at 2026-06-19 14:43:02 on dist-test-slave-cr22"
server_key: "a05496f4060b21214141f0a09c452140"
server_key_iv: "107a7722b98254dc17b1b659ddb1d67d"
server_key_version: "encryptionkey@0"
I20260619 14:43:02.885007 5228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "87626762ccf9441d8bf7cc58f0d98f28"
format_stamp: "Formatted at 2026-06-19 14:43:02 on dist-test-slave-cr22"
server_key: "a05496f4060b21214141f0a09c452140"
server_key_iv: "107a7722b98254dc17b1b659ddb1d67d"
server_key_version: "encryptionkey@0"
I20260619 14:43:02.888648 5228 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:43:02.891168 5242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:02.892568 5228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:43:02.892697 5228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "87626762ccf9441d8bf7cc58f0d98f28"
format_stamp: "Formatted at 2026-06-19 14:43:02 on dist-test-slave-cr22"
server_key: "a05496f4060b21214141f0a09c452140"
server_key_iv: "107a7722b98254dc17b1b659ddb1d67d"
server_key_version: "encryptionkey@0"
I20260619 14:43:02.892853 5228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:02.897528 5228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:02.898126 5228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:02.898305 5228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:02.898835 5228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:02.899793 5228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:02.899857 5228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:02.899923 5228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:02.899967 5228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:02.908816 5228 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35501
I20260619 14:43:02.908835 5355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35501 every 8 connection(s)
I20260619 14:43:02.909703 5228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:43:02.916213 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:02.911849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58805 (local address 127.0.98.62:34047)
0619 14:43:02.912014 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:02.912017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:02.913038 (+ 1021us) server_negotiation.cc:407] Connection header received
0619 14:43:02.914357 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:02.914360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:02.914405 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:02.914506 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:02.915673 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:02.915677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:02.915680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:02.915692 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:02.915715 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:02.915729 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:02.915899 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:02.916031 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 14:43:02.916076 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 14:43:02.917111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 5228
I20260619 14:43:02.917129 5358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:02.912233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34047 (local address 127.0.98.2:58805)
0619 14:43:02.912889 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:02.912928 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:02.914156 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:02.914660 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:02.914667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:02.914750 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:02.915283 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:43:02.915285 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:02.915358 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:43:02.915530 (+ 172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:02.915878 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:02.915887 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:02.915893 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:43:02.916074 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 14:43:02.916312 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":121,"threads_started":1}
I20260619 14:43:02.917248 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:43:02.917570 392 external_mini_cluster.cc:1428] Setting key 8a7ebcde2c210b0b6b6bda8ab66f0b6a
I20260619 14:43:02.918380 5356 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34047
I20260619 14:43:02.918617 5356 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:02.919094 5356 heartbeater.cc:511] Master 127.0.98.62:34047 requested a full tablet report, sending...
I20260619 14:43:02.919852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34047
--builtin_ntp_servers=127.0.98.20:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 14:43:02.920375 5037 ts_manager.cc:194] Registered new tserver with Master: 87626762ccf9441d8bf7cc58f0d98f28 (127.0.98.2:35501)
W20260619 14:43:03.035521 5359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:03.035821 5359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:03.035879 5359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:03.040084 5359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:03.040153 5359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:03.040211 5359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:43:03.045229 5359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:03.046594 5359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:03.048099 5359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:03.058418 5367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:03.058418 5365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:03.058444 5364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:03.058760 5359 server_base.cc:1034] running on GCE node
I20260619 14:43:03.059191 5359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:03.059983 5359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:03.061174 5359 hybrid_clock.cc:648] HybridClock initialized: now 1781880183061143 us; error 48 us; skew 500 ppm
I20260619 14:43:03.063060 5359 webserver.cc:460] Webserver started at http://127.0.98.3:35581/ using document root <none> and password file <none>
I20260619 14:43:03.063727 5359 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:03.063853 5359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:03.064081 5359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:03.066180 5359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "07750c51d42a408894a686eeff16e465"
format_stamp: "Formatted at 2026-06-19 14:43:03 on dist-test-slave-cr22"
server_key: "c3b56a74a51126762c3cc64371c16e5a"
server_key_iv: "a43e38a1f0e6f15a29800b39e9a97595"
server_key_version: "encryptionkey@0"
I20260619 14:43:03.066753 5359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "07750c51d42a408894a686eeff16e465"
format_stamp: "Formatted at 2026-06-19 14:43:03 on dist-test-slave-cr22"
server_key: "c3b56a74a51126762c3cc64371c16e5a"
server_key_iv: "a43e38a1f0e6f15a29800b39e9a97595"
server_key_version: "encryptionkey@0"
I20260619 14:43:03.071066 5359 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 14:43:03.073681 5373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:03.074980 5359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:43:03.075111 5359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "07750c51d42a408894a686eeff16e465"
format_stamp: "Formatted at 2026-06-19 14:43:03 on dist-test-slave-cr22"
server_key: "c3b56a74a51126762c3cc64371c16e5a"
server_key_iv: "a43e38a1f0e6f15a29800b39e9a97595"
server_key_version: "encryptionkey@0"
I20260619 14:43:03.075222 5359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:03.079892 5359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:03.080504 5359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:03.080684 5359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:03.081218 5359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:03.082166 5359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:03.082234 5359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:03.082297 5359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:03.082346 5359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:03.091194 5359 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36183
I20260619 14:43:03.091238 5486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36183 every 8 connection(s)
I20260619 14:43:03.092156 5359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:43:03.096493 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 5359
I20260619 14:43:03.096674 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:43:03.096956 392 external_mini_cluster.cc:1428] Setting key e99f405e8f3b0c5c0616ec695beb4470
I20260619 14:43:03.099613 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:03.094124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37643 (local address 127.0.98.62:34047)
0619 14:43:03.094262 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:03.094266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:03.095317 (+ 1051us) server_negotiation.cc:407] Connection header received
0619 14:43:03.096913 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:03.096917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:03.096989 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:03.097099 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:03.098525 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:03.098528 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:03.098530 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:03.098542 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:03.098563 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:03.098572 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:03.098646 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:03.099428 (+ 782us) server_negotiation.cc:299] Negotiation successful
0619 14:43:03.099505 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:43:03.100533 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:03.094512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34047 (local address 127.0.98.3:37643)
0619 14:43:03.095186 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:03.095227 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:03.096601 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:03.097323 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:03.097341 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:03.097460 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:03.098203 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:43:03.098205 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:03.098262 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:43:03.098361 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:03.099208 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:03.099217 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:03.099226 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 14:43:03.099391 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 14:43:03.099693 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":116,"threads_started":1}
I20260619 14:43:03.101966 5487 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34047
I20260619 14:43:03.102226 5487 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:03.102648 5487 heartbeater.cc:511] Master 127.0.98.62:34047 requested a full tablet report, sending...
I20260619 14:43:03.103808 5037 ts_manager.cc:194] Registered new tserver with Master: 07750c51d42a408894a686eeff16e465 (127.0.98.3:36183)
I20260619 14:43:03.111729 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:43:03.115526 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:03.113655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.0.98.62:34047)
0619 14:43:03.113817 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:03.113821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:03.113924 (+ 103us) server_negotiation.cc:407] Connection header received
0619 14:43:03.114054 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:03.114057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:03.114094 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:03.114174 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:03.115082 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:03.115085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:03.115086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:03.115097 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:03.115122 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:03.115130 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:03.115208 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:03.115284 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 14:43:03.115318 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:43:03.119076 5037 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:43:03.122437 5037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:43:03.134869 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:03.131593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39919 (local address 127.0.0.1:51524)
0619 14:43:03.132173 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:03.132184 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:03.132240 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:03.133316 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:03.133318 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:03.133341 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:03.133911 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:43:03.133916 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:03.133939 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:43:03.134015 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:03.134502 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:03.134523 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:03.134530 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:43:03.134647 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 14:43:03.134718 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":105,"threads_started":1}
I20260619 14:43:03.134950 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:03.132210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51524 (local address 127.0.98.1:39919)
0619 14:43:03.132934 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:03.132946 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:03.132968 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:43:03.133019 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:03.133024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:03.133185 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:03.133326 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:03.134157 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:03.134163 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:03.134177 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:03.134338 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:03.134383 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:03.134408 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:03.134517 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:03.134670 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 14:43:03.134747 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":120,"threads_started":1}
I20260619 14:43:03.135057 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:03.132517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33662 (local address 127.0.98.2:35501)
0619 14:43:03.132934 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:03.132946 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:03.132968 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:43:03.133018 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:03.133024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:03.133185 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:03.133326 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:03.134334 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:03.134343 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:03.134357 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:03.134374 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:03.134418 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:03.134448 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:03.134564 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:03.134800 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 14:43:03.134876 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":115,"threads_started":1}
I20260619 14:43:03.135293 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:03.132231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35501 (local address 127.0.0.1:33662)
0619 14:43:03.132594 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:03.132609 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:03.132696 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:03.133523 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:03.133528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:03.133541 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:03.134170 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:43:03.134172 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:03.134182 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:43:03.134206 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:03.134570 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:03.134573 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:03.134575 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 14:43:03.134647 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 14:43:03.134728 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":6,"thread_start_us":101,"threads_started":1}
I20260619 14:43:03.135629 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:03.131533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36183 (local address 127.0.0.1:39338)
0619 14:43:03.132047 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:03.132096 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:03.132202 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:03.133160 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:03.133166 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:03.133238 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:03.134225 (+ 987us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:43:03.134229 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:03.134240 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:43:03.134264 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:03.134876 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:03.134879 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:03.134882 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:43:03.134938 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 14:43:03.135517 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":77,"threads_started":1}
I20260619 14:43:03.136163 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:03.131678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39338 (local address 127.0.98.3:36183)
0619 14:43:03.132683 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:03.132696 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:03.132721 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:43:03.132777 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:03.132783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:03.132992 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:03.133997 (+ 1005us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:03.134669 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:03.134673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:03.134684 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:03.134693 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:03.134728 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:03.134749 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:03.135878 (+ 1129us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:03.135927 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 14:43:03.135991 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":893,"thread_start_us":112,"threads_started":1}
I20260619 14:43:03.138370 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 9421dc6a7c034eda9ff3d68d062a57f4 (DEFAULT_TABLE table=test-table [id=c929896fdc494e14adaad26a9f18c859]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:03.138240 5159 tablet_service.cc:1467] Processing CreateTablet for tablet 9421dc6a7c034eda9ff3d68d062a57f4 (DEFAULT_TABLE table=test-table [id=c929896fdc494e14adaad26a9f18c859]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:03.138240 5290 tablet_service.cc:1467] Processing CreateTablet for tablet 9421dc6a7c034eda9ff3d68d062a57f4 (DEFAULT_TABLE table=test-table [id=c929896fdc494e14adaad26a9f18c859]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:03.139320 5159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9421dc6a7c034eda9ff3d68d062a57f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:03.139320 5290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9421dc6a7c034eda9ff3d68d062a57f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:03.139336 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9421dc6a7c034eda9ff3d68d062a57f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:03.145382 5504 tablet_bootstrap.cc:492] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465: Bootstrap starting.
I20260619 14:43:03.146986 5505 tablet_bootstrap.cc:492] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28: Bootstrap starting.
I20260619 14:43:03.146986 5506 tablet_bootstrap.cc:492] T 9421dc6a7c034eda9ff3d68d062a57f4 P 8a82737d751b4ccd8a0c5ac180625603: Bootstrap starting.
I20260619 14:43:03.147701 5504 tablet_bootstrap.cc:654] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:03.148720 5504 log.cc:824] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:03.149111 5506 tablet_bootstrap.cc:654] T 9421dc6a7c034eda9ff3d68d062a57f4 P 8a82737d751b4ccd8a0c5ac180625603: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:03.149111 5505 tablet_bootstrap.cc:654] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:03.149932 5506 log.cc:824] T 9421dc6a7c034eda9ff3d68d062a57f4 P 8a82737d751b4ccd8a0c5ac180625603: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:03.149932 5505 log.cc:824] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:03.150697 5504 tablet_bootstrap.cc:492] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465: No bootstrap required, opened a new log
I20260619 14:43:03.150924 5504 ts_tablet_manager.cc:1397] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:43:03.151943 5505 tablet_bootstrap.cc:492] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28: No bootstrap required, opened a new log
I20260619 14:43:03.151943 5506 tablet_bootstrap.cc:492] T 9421dc6a7c034eda9ff3d68d062a57f4 P 8a82737d751b4ccd8a0c5ac180625603: No bootstrap required, opened a new log
I20260619 14:43:03.152127 5506 ts_tablet_manager.cc:1397] T 9421dc6a7c034eda9ff3d68d062a57f4 P 8a82737d751b4ccd8a0c5ac180625603: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:43:03.152127 5505 ts_tablet_manager.cc:1397] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:43:03.154855 5504 raft_consensus.cc:348] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07750c51d42a408894a686eeff16e465" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36183 } } peers { permanent_uuid: "8a82737d751b4ccd8a0c5ac180625603" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39919 } } peers { permanent_uuid: "87626762ccf9441d8bf7cc58f0d98f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } }
I20260619 14:43:03.155086 5504 raft_consensus.cc:374] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:03.155155 5504 raft_consensus.cc:729] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07750c51d42a408894a686eeff16e465, State: Initialized, Role: FOLLOWER
I20260619 14:43:03.155617 5504 consensus_queue.cc:260] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07750c51d42a408894a686eeff16e465" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36183 } } peers { permanent_uuid: "8a82737d751b4ccd8a0c5ac180625603" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39919 } } peers { permanent_uuid: "87626762ccf9441d8bf7cc58f0d98f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } }
I20260619 14:43:03.156462 5504 ts_tablet_manager.cc:1428] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:43:03.156605 5506 raft_consensus.cc:348] T 9421dc6a7c034eda9ff3d68d062a57f4 P 8a82737d751b4ccd8a0c5ac180625603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07750c51d42a408894a686eeff16e465" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36183 } } peers { permanent_uuid: "8a82737d751b4ccd8a0c5ac180625603" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39919 } } peers { permanent_uuid: "87626762ccf9441d8bf7cc58f0d98f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } }
I20260619 14:43:03.156605 5505 raft_consensus.cc:348] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07750c51d42a408894a686eeff16e465" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36183 } } peers { permanent_uuid: "8a82737d751b4ccd8a0c5ac180625603" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39919 } } peers { permanent_uuid: "87626762ccf9441d8bf7cc58f0d98f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } }
I20260619 14:43:03.156821 5487 heartbeater.cc:503] Master 127.0.98.62:34047 was elected leader, sending a full tablet report...
I20260619 14:43:03.156859 5505 raft_consensus.cc:374] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:03.156915 5505 raft_consensus.cc:729] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87626762ccf9441d8bf7cc58f0d98f28, State: Initialized, Role: FOLLOWER
I20260619 14:43:03.156925 5506 raft_consensus.cc:374] T 9421dc6a7c034eda9ff3d68d062a57f4 P 8a82737d751b4ccd8a0c5ac180625603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:03.156975 5506 raft_consensus.cc:729] T 9421dc6a7c034eda9ff3d68d062a57f4 P 8a82737d751b4ccd8a0c5ac180625603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a82737d751b4ccd8a0c5ac180625603, State: Initialized, Role: FOLLOWER
I20260619 14:43:03.157351 5506 consensus_queue.cc:260] T 9421dc6a7c034eda9ff3d68d062a57f4 P 8a82737d751b4ccd8a0c5ac180625603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07750c51d42a408894a686eeff16e465" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36183 } } peers { permanent_uuid: "8a82737d751b4ccd8a0c5ac180625603" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39919 } } peers { permanent_uuid: "87626762ccf9441d8bf7cc58f0d98f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } }
I20260619 14:43:03.157346 5505 consensus_queue.cc:260] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07750c51d42a408894a686eeff16e465" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36183 } } peers { permanent_uuid: "8a82737d751b4ccd8a0c5ac180625603" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39919 } } peers { permanent_uuid: "87626762ccf9441d8bf7cc58f0d98f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } }
I20260619 14:43:03.158270 5505 ts_tablet_manager.cc:1428] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:43:03.158270 5506 ts_tablet_manager.cc:1428] T 9421dc6a7c034eda9ff3d68d062a57f4 P 8a82737d751b4ccd8a0c5ac180625603: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260619 14:43:03.158613 5356 heartbeater.cc:503] Master 127.0.98.62:34047 was elected leader, sending a full tablet report...
I20260619 14:43:03.158672 5225 heartbeater.cc:503] Master 127.0.98.62:34047 was elected leader, sending a full tablet report...
W20260619 14:43:03.160576 5357 tablet.cc:2369] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:43:03.230677 5226 tablet.cc:2369] T 9421dc6a7c034eda9ff3d68d062a57f4 P 8a82737d751b4ccd8a0c5ac180625603: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:43:03.292016 5512 raft_consensus.cc:482] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:43:03.292236 5512 raft_consensus.cc:504] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07750c51d42a408894a686eeff16e465" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36183 } } peers { permanent_uuid: "8a82737d751b4ccd8a0c5ac180625603" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39919 } } peers { permanent_uuid: "87626762ccf9441d8bf7cc58f0d98f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } }
I20260619 14:43:03.293437 5512 leader_election.cc:288] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07750c51d42a408894a686eeff16e465 (127.0.98.3:36183), 8a82737d751b4ccd8a0c5ac180625603 (127.0.98.1:39919)
I20260619 14:43:03.295563 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:03.293953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47345 (local address 127.0.98.1:39919)
0619 14:43:03.294102 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:03.294107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:03.294121 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:43:03.294168 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:03.294171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:03.294217 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:03.294307 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:03.295082 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:03.295086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:03.295089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:03.295107 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:03.295126 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:03.295134 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:03.295209 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:03.295351 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 14:43:03.295396 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:43:03.295555 5358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:03.293732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39919 (local address 127.0.98.2:47345)
0619 14:43:03.293859 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:03.293878 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:03.293966 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:03.294377 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:03.294381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:03.294396 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:03.294932 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:43:03.294934 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:03.294943 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:43:03.294968 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:03.295234 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:03.295246 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:03.295249 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:43:03.295336 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:43:03.295374 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 14:43:03.296065 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:03.293906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50291 (local address 127.0.98.3:36183)
0619 14:43:03.294045 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:03.294049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:03.294338 (+ 289us) server_negotiation.cc:407] Connection header received
0619 14:43:03.294391 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:03.294394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:03.294437 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:03.294550 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:03.295406 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:03.295410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:03.295412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:03.295424 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:03.295471 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:03.295482 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:03.295566 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:03.295732 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 14:43:03.295781 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:43:03.296397 5179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9421dc6a7c034eda9ff3d68d062a57f4" candidate_uuid: "87626762ccf9441d8bf7cc58f0d98f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a82737d751b4ccd8a0c5ac180625603" is_pre_election: true
I20260619 14:43:03.296475 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:03.293732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36183 (local address 127.0.98.2:50291)
0619 14:43:03.294215 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:03.294229 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:03.294304 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:03.294595 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:03.294601 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:03.294619 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:03.295201 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:43:03.295205 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:03.295219 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:43:03.295242 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:03.295589 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:03.295593 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:03.295600 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:43:03.296294 (+ 694us) client_negotiation.cc:240] Negotiation successful
0619 14:43:03.296351 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":105,"thread_start_us":155,"threads_started":1}
I20260619 14:43:03.296720 5179 raft_consensus.cc:2454] T 9421dc6a7c034eda9ff3d68d062a57f4 P 8a82737d751b4ccd8a0c5ac180625603 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87626762ccf9441d8bf7cc58f0d98f28 in term 0.
I20260619 14:43:03.297124 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9421dc6a7c034eda9ff3d68d062a57f4" candidate_uuid: "87626762ccf9441d8bf7cc58f0d98f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07750c51d42a408894a686eeff16e465" is_pre_election: true
I20260619 14:43:03.297212 5244 leader_election.cc:302] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87626762ccf9441d8bf7cc58f0d98f28, 8a82737d751b4ccd8a0c5ac180625603; no voters:
I20260619 14:43:03.297415 5441 raft_consensus.cc:2454] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87626762ccf9441d8bf7cc58f0d98f28 in term 0.
I20260619 14:43:03.297506 5512 raft_consensus.cc:2781] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:43:03.297593 5512 raft_consensus.cc:482] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:43:03.297649 5512 raft_consensus.cc:3037] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:03.298724 5512 raft_consensus.cc:504] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07750c51d42a408894a686eeff16e465" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36183 } } peers { permanent_uuid: "8a82737d751b4ccd8a0c5ac180625603" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39919 } } peers { permanent_uuid: "87626762ccf9441d8bf7cc58f0d98f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } }
I20260619 14:43:03.299055 5512 leader_election.cc:288] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [CANDIDATE]: Term 1 election: Requested vote from peers 07750c51d42a408894a686eeff16e465 (127.0.98.3:36183), 8a82737d751b4ccd8a0c5ac180625603 (127.0.98.1:39919)
I20260619 14:43:03.299392 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9421dc6a7c034eda9ff3d68d062a57f4" candidate_uuid: "87626762ccf9441d8bf7cc58f0d98f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07750c51d42a408894a686eeff16e465"
I20260619 14:43:03.299424 5179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9421dc6a7c034eda9ff3d68d062a57f4" candidate_uuid: "87626762ccf9441d8bf7cc58f0d98f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a82737d751b4ccd8a0c5ac180625603"
I20260619 14:43:03.299568 5441 raft_consensus.cc:3037] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:03.299618 5179 raft_consensus.cc:3037] T 9421dc6a7c034eda9ff3d68d062a57f4 P 8a82737d751b4ccd8a0c5ac180625603 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:03.300581 5441 raft_consensus.cc:2454] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87626762ccf9441d8bf7cc58f0d98f28 in term 1.
I20260619 14:43:03.300743 5179 raft_consensus.cc:2454] T 9421dc6a7c034eda9ff3d68d062a57f4 P 8a82737d751b4ccd8a0c5ac180625603 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87626762ccf9441d8bf7cc58f0d98f28 in term 1.
I20260619 14:43:03.300869 5246 leader_election.cc:302] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07750c51d42a408894a686eeff16e465, 87626762ccf9441d8bf7cc58f0d98f28; no voters:
I20260619 14:43:03.301074 5512 raft_consensus.cc:2781] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:03.301319 5512 raft_consensus.cc:686] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [term 1 LEADER]: Becoming Leader. State: Replica: 87626762ccf9441d8bf7cc58f0d98f28, State: Running, Role: LEADER
I20260619 14:43:03.301653 5512 consensus_queue.cc:237] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07750c51d42a408894a686eeff16e465" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36183 } } peers { permanent_uuid: "8a82737d751b4ccd8a0c5ac180625603" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39919 } } peers { permanent_uuid: "87626762ccf9441d8bf7cc58f0d98f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } }
I20260619 14:43:03.305168 5035 catalog_manager.cc:5495] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87626762ccf9441d8bf7cc58f0d98f28 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "87626762ccf9441d8bf7cc58f0d98f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07750c51d42a408894a686eeff16e465" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a82737d751b4ccd8a0c5ac180625603" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87626762ccf9441d8bf7cc58f0d98f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35501 } health_report { overall_health: HEALTHY } } }
I20260619 14:43:03.325316 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:03.324012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37800 (local address 127.0.98.2:35501)
0619 14:43:03.324151 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:03.324154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:03.324167 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:43:03.324238 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:03.324241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:03.324296 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:03.324372 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:03.325013 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:03.325016 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:03.325018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:03.325028 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:03.325045 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:03.325053 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:03.325125 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:03.325181 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 14:43:03.325212 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:43:03.331060 5179 raft_consensus.cc:1261] T 9421dc6a7c034eda9ff3d68d062a57f4 P 8a82737d751b4ccd8a0c5ac180625603 [term 1 FOLLOWER]: Refusing update from remote peer 87626762ccf9441d8bf7cc58f0d98f28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:03.331089 5441 raft_consensus.cc:1261] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465 [term 1 FOLLOWER]: Refusing update from remote peer 87626762ccf9441d8bf7cc58f0d98f28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:03.331766 5512 consensus_queue.cc:1035] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07750c51d42a408894a686eeff16e465" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:03.331983 5514 consensus_queue.cc:1035] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a82737d751b4ccd8a0c5ac180625603" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:03.340147 5517 mvcc.cc:205] Tried to move back new op lower bound from 7298581230916988928 to 7298581230805778432. Current Snapshot: MvccSnapshot[applied={T|T < 7298581230916988928}]
I20260619 14:43:03.340327 5518 mvcc.cc:205] Tried to move back new op lower bound from 7298581230916988928 to 7298581230805778432. Current Snapshot: MvccSnapshot[applied={T|T < 7298581230916988928}]
I20260619 14:43:03.340790 5519 mvcc.cc:205] Tried to move back new op lower bound from 7298581230916988928 to 7298581230805778432. Current Snapshot: MvccSnapshot[applied={T|T < 7298581230916988928}]
I20260619 14:43:03.348579 5037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:43:03.348841 5037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:43:03.351517 5037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b52b524adde46d1ba1c8966e81d6f60: Sending DeleteTablet for 3 replicas of tablet 9421dc6a7c034eda9ff3d68d062a57f4
I20260619 14:43:03.352360 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 9421dc6a7c034eda9ff3d68d062a57f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:03 UTC) from {username='slave'} at 127.0.0.1:39338
I20260619 14:43:03.352348 5159 tablet_service.cc:1514] Processing DeleteTablet for tablet 9421dc6a7c034eda9ff3d68d062a57f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:03 UTC) from {username='slave'} at 127.0.0.1:51524
I20260619 14:43:03.352867 5527 tablet_replica.cc:330] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465: stopping tablet replica
I20260619 14:43:03.353047 5290 tablet_service.cc:1514] Processing DeleteTablet for tablet 9421dc6a7c034eda9ff3d68d062a57f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:03 UTC) from {username='slave'} at 127.0.0.1:33662
I20260619 14:43:03.353330 5527 raft_consensus.cc:2229] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:43:03.353500 5529 tablet_replica.cc:330] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28: stopping tablet replica
I20260619 14:43:03.353544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 5097
I20260619 14:43:03.354473 5527 raft_consensus.cc:2258] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:03.355587 5529 raft_consensus.cc:2229] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:43:03.356005 5529 raft_consensus.cc:2258] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:03.356299 5527 ts_tablet_manager.cc:1905] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:03.357378 5529 ts_tablet_manager.cc:1905] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:03.359843 5527 ts_tablet_manager.cc:1918] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:43:03.360031 5527 log.cc:1196] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/9421dc6a7c034eda9ff3d68d062a57f4
I20260619 14:43:03.360059 5529 ts_tablet_manager.cc:1918] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:43:03.360152 5529 log.cc:1196] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/9421dc6a7c034eda9ff3d68d062a57f4
I20260619 14:43:03.360494 5529 ts_tablet_manager.cc:1939] T 9421dc6a7c034eda9ff3d68d062a57f4 P 87626762ccf9441d8bf7cc58f0d98f28: Deleting consensus metadata
I20260619 14:43:03.360523 5527 ts_tablet_manager.cc:1939] T 9421dc6a7c034eda9ff3d68d062a57f4 P 07750c51d42a408894a686eeff16e465: Deleting consensus metadata
I20260619 14:43:03.361666 5024 catalog_manager.cc:4841] TS 87626762ccf9441d8bf7cc58f0d98f28 (127.0.98.2:35501): tablet 9421dc6a7c034eda9ff3d68d062a57f4 (table test-table [id=c929896fdc494e14adaad26a9f18c859]) successfully deleted
I20260619 14:43:03.361986 5024 catalog_manager.cc:4841] TS 07750c51d42a408894a686eeff16e465 (127.0.98.3:36183): tablet 9421dc6a7c034eda9ff3d68d062a57f4 (table test-table [id=c929896fdc494e14adaad26a9f18c859]) successfully deleted
W20260619 14:43:03.362879 5022 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39919 (error 108)
I20260619 14:43:03.363041 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 5228
I20260619 14:43:03.368299 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:03.367991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39919 (local address 127.0.0.1:37494)
0619 14:43:03.368114 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:03.368207 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 14:43:03.368537 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 5359
W20260619 14:43:03.368577 5022 catalog_manager.cc:4584] TS 8a82737d751b4ccd8a0c5ac180625603 (127.0.98.1:39919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9421dc6a7c034eda9ff3d68d062a57f4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39919: connect: Connection refused (error 111)
I20260619 14:43:03.374178 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 5006
2026-06-19T14:43:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1084 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 14:43:03.395907 392 mini_oidc.cc:152] Starting JWKS server
I20260619 14:43:03.397523 392 webserver.cc:460] Webserver started at https://127.0.0.1:44679/ using document root <none> and password file <none>
I20260619 14:43:03.397708 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:43:03.397979 392 webserver.cc:460] Webserver started at http://127.0.0.1:36215/ using document root <none> and password file <none>
2026-06-19T14:43:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:43:03Z Disabled control of system clock
I20260619 14:43:03.407016 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33213
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33213
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44679/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0/testchainca.cert with env {}
W20260619 14:43:03.516031 5536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:03.516291 5536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:03.516340 5536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:03.520404 5536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:43:03.520468 5536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:03.520484 5536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:03.520500 5536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:43:03.520515 5536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:43:03.520576 5536 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:43:03.520596 5536 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44679/jwks.json
I20260619 14:43:03.526015 5536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33213
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44679/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:03.527128 5536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:03.528173 5536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:03.537456 5542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:03.537509 5541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:03.537595 5544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:03.537793 5536 server_base.cc:1034] running on GCE node
I20260619 14:43:03.538513 5536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:03.539891 5536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:03.541097 5536 hybrid_clock.cc:648] HybridClock initialized: now 1781880183541076 us; error 39 us; skew 500 ppm
I20260619 14:43:03.543356 5536 webserver.cc:460] Webserver started at http://127.0.98.62:37675/ using document root <none> and password file <none>
I20260619 14:43:03.544034 5536 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:03.544100 5536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:03.544348 5536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:03.546576 5536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "6b950b69e63a46e19d67b1ac0b93bbda"
format_stamp: "Formatted at 2026-06-19 14:43:03 on dist-test-slave-cr22"
server_key: "069c5a4e4c07b5155be2f2ccf619b423"
server_key_iv: "d4f11d1d48c91fdf93389ca52d0ac8dd"
server_key_version: "encryptionkey@0"
I20260619 14:43:03.547164 5536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "6b950b69e63a46e19d67b1ac0b93bbda"
format_stamp: "Formatted at 2026-06-19 14:43:03 on dist-test-slave-cr22"
server_key: "069c5a4e4c07b5155be2f2ccf619b423"
server_key_iv: "d4f11d1d48c91fdf93389ca52d0ac8dd"
server_key_version: "encryptionkey@0"
I20260619 14:43:03.550984 5536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 14:43:03.553848 5550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:03.555297 5536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:43:03.555495 5536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "6b950b69e63a46e19d67b1ac0b93bbda"
format_stamp: "Formatted at 2026-06-19 14:43:03 on dist-test-slave-cr22"
server_key: "069c5a4e4c07b5155be2f2ccf619b423"
server_key_iv: "d4f11d1d48c91fdf93389ca52d0ac8dd"
server_key_version: "encryptionkey@0"
I20260619 14:43:03.555639 5536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:03.580909 5536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:03.581629 5536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:03.581827 5536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:03.589047 5536 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33213
I20260619 14:43:03.589062 5602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33213 every 8 connection(s)
I20260619 14:43:03.590000 5536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:43:03.592903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 5536
I20260619 14:43:03.593039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:43:03.593304 392 external_mini_cluster.cc:1428] Setting key 2cb67064662d9f3f71c8d8e6dc339e09
I20260619 14:43:03.593694 5603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:03.600841 5603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda: Bootstrap starting.
I20260619 14:43:03.602960 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:03.594754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48080 (local address 127.0.98.62:33213)
0619 14:43:03.595383 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:03.595402 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:03.595460 (+ 58us) server_negotiation.cc:407] Connection header received
0619 14:43:03.596371 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:03.596389 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:03.596884 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:03.597169 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:03.597552 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:03.598539 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:03.599794 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:03.600252 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:03.600385 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:03.601313 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:03.601318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:03.601351 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:03.601728 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:03.601778 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:03.601797 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:03.601904 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:03.602021 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 14:43:03.602268 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":137,"threads_started":1}
I20260619 14:43:03.603092 5603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:03.603919 5603 log.cc:824] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:03.605688 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:43:03.605944 5603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda: No bootstrap required, opened a new log
I20260619 14:43:03.609812 5603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b950b69e63a46e19d67b1ac0b93bbda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33213 } }
I20260619 14:43:03.610025 5603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:03.610098 5603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b950b69e63a46e19d67b1ac0b93bbda, State: Initialized, Role: FOLLOWER
I20260619 14:43:03.610523 5603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b950b69e63a46e19d67b1ac0b93bbda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33213 } }
I20260619 14:43:03.610646 5603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:43:03.610708 5603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:43:03.610824 5603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:03.611763 5603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b950b69e63a46e19d67b1ac0b93bbda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33213 } }
I20260619 14:43:03.612087 5603 leader_election.cc:302] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b950b69e63a46e19d67b1ac0b93bbda; no voters:
I20260619 14:43:03.612377 5603 leader_election.cc:288] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:43:03.612489 5608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:03.612736 5608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda [term 1 LEADER]: Becoming Leader. State: Replica: 6b950b69e63a46e19d67b1ac0b93bbda, State: Running, Role: LEADER
I20260619 14:43:03.613106 5608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b950b69e63a46e19d67b1ac0b93bbda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33213 } }
I20260619 14:43:03.613682 5603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:43:03.614748 5610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b950b69e63a46e19d67b1ac0b93bbda. Latest consensus state: current_term: 1 leader_uuid: "6b950b69e63a46e19d67b1ac0b93bbda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b950b69e63a46e19d67b1ac0b93bbda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33213 } } }
I20260619 14:43:03.614874 5610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:03.615139 5609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b950b69e63a46e19d67b1ac0b93bbda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b950b69e63a46e19d67b1ac0b93bbda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33213 } } }
I20260619 14:43:03.615231 5609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:03.615272 5617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:43:03.618552 5617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:43:03.624519 5617 catalog_manager.cc:1347] Generated new cluster ID: 83b17d69b6074d08bc8b7808c17a7efa
I20260619 14:43:03.624594 5617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:43:03.636219 5617 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:43:03.637140 5617 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:43:03.641944 5617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b950b69e63a46e19d67b1ac0b93bbda: Generated new TSK 0
I20260619 14:43:03.642608 5617 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 14:43:03.672431 5633 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 14:43:03.672567 5633 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 14:43:03.673135 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:03.655464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.0.98.62:33213)
0619 14:43:03.655588 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:03.655593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:03.655770 (+ 177us) server_negotiation.cc:407] Connection header received
0619 14:43:03.655968 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:03.655971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:03.656026 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:03.656137 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:43:03.656419 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:03.657240 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:03.658253 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:03.658589 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:03.658700 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:03.659026 (+ 326us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:43:03.672727 (+ 13701us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 14:43:03.672997 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48090: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 14:43:03.673213 5606 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48090: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 14:43:03.674288 5632 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:03.655305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33213 (local address 127.0.0.1:48090)
0619 14:43:03.655689 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:03.655702 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:03.655813 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:03.656172 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:03.656174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:03.656181 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:43:03.656315 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:03.656321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:03.657384 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:03.657387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:03.658115 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:03.658124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:03.658691 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:03.658713 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:03.658789 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:03.658835 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:43:03.672949 (+ 14114us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 14:43:03.673154 (+ 205us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33213: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":137,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:33213: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 14:43:03.675073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 5536
2026-06-19T14:43:03Z chronyd exiting
I20260619 14:43:03.712459 392 test_util.cc:175] -----------------------------------------------
I20260619 14:43:03.712546 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880155260304-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (318 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 14:43:03.713639 392 mini_oidc.cc:152] Starting JWKS server
I20260619 14:43:03.714445 392 webserver.cc:460] Webserver started at https://127.0.0.1:45035/ using document root <none> and password file <none>
I20260619 14:43:03.714521 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:43:03.714779 392 webserver.cc:460] Webserver started at http://127.0.0.1:42409/ using document root <none> and password file <none>
2026-06-19T14:43:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:43:03Z Disabled control of system clock
I20260619 14:43:03.724212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43863
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43863
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45035/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880155260304-392-0/testchainca.cert with env {}
W20260619 14:43:03.840798 5639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:03.841084 5639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:03.841125 5639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:03.845635 5639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:43:03.845695 5639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:03.845710 5639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:03.845727 5639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:43:03.845742 5639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:43:03.845758 5639 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:43:03.845772 5639 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45035/jwks.json
I20260619 14:43:03.851229 5639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45035/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880155260304-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:03.852370 5639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:03.853338 5639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:03.862250 5644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:03.862219 5645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:03.862399 5639 server_base.cc:1034] running on GCE node
W20260619 14:43:03.862358 5647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:03.863116 5639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:03.864317 5639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:03.865506 5639 hybrid_clock.cc:648] HybridClock initialized: now 1781880183865492 us; error 24 us; skew 500 ppm
I20260619 14:43:03.867406 5639 webserver.cc:460] Webserver started at http://127.0.98.62:41529/ using document root <none> and password file <none>
I20260619 14:43:03.868031 5639 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:03.868117 5639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:03.868338 5639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:03.870353 5639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "b4c608d841bb439985022e8f7ccbbae6"
format_stamp: "Formatted at 2026-06-19 14:43:03 on dist-test-slave-cr22"
server_key: "e1054b314397d4ae2209d7d9e372abc8"
server_key_iv: "e13e6791ae800fc1782b5580cbe5820c"
server_key_version: "encryptionkey@0"
I20260619 14:43:03.870916 5639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "b4c608d841bb439985022e8f7ccbbae6"
format_stamp: "Formatted at 2026-06-19 14:43:03 on dist-test-slave-cr22"
server_key: "e1054b314397d4ae2209d7d9e372abc8"
server_key_iv: "e13e6791ae800fc1782b5580cbe5820c"
server_key_version: "encryptionkey@0"
I20260619 14:43:03.874550 5639 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:43:03.877103 5653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:03.878393 5639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:43:03.878530 5639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "b4c608d841bb439985022e8f7ccbbae6"
format_stamp: "Formatted at 2026-06-19 14:43:03 on dist-test-slave-cr22"
server_key: "e1054b314397d4ae2209d7d9e372abc8"
server_key_iv: "e13e6791ae800fc1782b5580cbe5820c"
server_key_version: "encryptionkey@0"
I20260619 14:43:03.878643 5639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:03.897102 5639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:03.897879 5639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:03.898079 5639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:03.905506 5639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43863
I20260619 14:43:03.905606 5705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43863 every 8 connection(s)
I20260619 14:43:03.906575 5639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:43:03.910140 5706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:03.915210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 5639
I20260619 14:43:03.915362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:43:03.915632 392 external_mini_cluster.cc:1428] Setting key cb2f611b69bdfe840823fdf3c95881e2
I20260619 14:43:03.916317 5706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6: Bootstrap starting.
I20260619 14:43:03.918864 5706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:03.919811 5706 log.cc:824] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:03.922196 5706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6: No bootstrap required, opened a new log
I20260619 14:43:03.925287 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:03.917062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60440 (local address 127.0.98.62:43863)
0619 14:43:03.917709 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:03.917725 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:03.917762 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:43:03.918392 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:03.918408 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:03.918791 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:03.919103 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:03.919505 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:03.920681 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:03.921883 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:03.922389 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:03.922513 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:03.923363 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:03.923368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:03.923381 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:03.923910 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:03.923967 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:03.923987 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:03.924069 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:03.924227 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 14:43:03.924520 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":160,"threads_started":1}
I20260619 14:43:03.926272 5706 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c608d841bb439985022e8f7ccbbae6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43863 } }
I20260619 14:43:03.926510 5706 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:03.926591 5706 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c608d841bb439985022e8f7ccbbae6, State: Initialized, Role: FOLLOWER
I20260619 14:43:03.927024 5706 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c608d841bb439985022e8f7ccbbae6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43863 } }
I20260619 14:43:03.927201 5706 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:43:03.927271 5706 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:43:03.927368 5706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:03.928117 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:43:03.981472 5706 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c608d841bb439985022e8f7ccbbae6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43863 } }
I20260619 14:43:03.982069 5706 leader_election.cc:302] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4c608d841bb439985022e8f7ccbbae6; no voters:
I20260619 14:43:03.982589 5706 leader_election.cc:288] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:43:03.982786 5711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:03.983078 5711 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6 [term 1 LEADER]: Becoming Leader. State: Replica: b4c608d841bb439985022e8f7ccbbae6, State: Running, Role: LEADER
I20260619 14:43:03.983418 5711 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c608d841bb439985022e8f7ccbbae6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43863 } }
I20260619 14:43:03.984319 5706 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:43:03.985240 5712 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4c608d841bb439985022e8f7ccbbae6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c608d841bb439985022e8f7ccbbae6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43863 } } }
I20260619 14:43:03.985396 5712 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:03.985464 5713 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4c608d841bb439985022e8f7ccbbae6. Latest consensus state: current_term: 1 leader_uuid: "b4c608d841bb439985022e8f7ccbbae6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c608d841bb439985022e8f7ccbbae6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43863 } } }
I20260619 14:43:03.985576 5713 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:03.986163 5718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:43:03.989878 5718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:43:03.996376 5718 catalog_manager.cc:1347] Generated new cluster ID: 72085d27f1184dd68ca85e57c18b9d4d
I20260619 14:43:03.996466 5718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:43:04.059091 5718 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:43:04.060400 5718 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:43:04.071698 5718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4c608d841bb439985022e8f7ccbbae6: Generated new TSK 0
I20260619 14:43:04.072651 5718 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 14:43:04.114960 5736 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 14:43:04.115064 5736 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 14:43:04.115862 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:04.097653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60456 (local address 127.0.98.62:43863)
0619 14:43:04.097925 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:04.097932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:04.098053 (+ 121us) server_negotiation.cc:407] Connection header received
0619 14:43:04.098249 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:04.098254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:04.098318 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:04.098449 (+ 131us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:43:04.098741 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:04.099589 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:04.100569 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:04.100912 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:04.101017 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:04.101366 (+ 349us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:43:04.115359 (+ 13993us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 14:43:04.115702 (+ 343us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60456: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":139}
W20260619 14:43:04.115909 5735 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:04.097507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43863 (local address 127.0.0.1:60456)
0619 14:43:04.097964 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:04.097977 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:04.098079 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:04.098478 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:04.098481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:04.098489 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:43:04.098588 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:04.098593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:04.099734 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:04.099737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:04.100453 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:04.100462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:04.101060 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:04.101081 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:04.101152 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:04.101177 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:43:04.115612 (+ 14435us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 14:43:04.115767 (+ 155us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43863: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":137,"threads_started":1}
W20260619 14:43:04.115942 5709 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60456: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260619 14:43:04.116722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 5639
2026-06-19T14:43:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (443 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 14:43:04.156718 392 mini_oidc.cc:152] Starting JWKS server
I20260619 14:43:04.157579 392 webserver.cc:460] Webserver started at https://127.0.0.1:34483/ using document root <none> and password file <none>
I20260619 14:43:04.157682 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:43:04.157948 392 webserver.cc:460] Webserver started at http://127.0.0.1:36437/ using document root <none> and password file <none>
2026-06-19T14:43:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:43:04Z Disabled control of system clock
I20260619 14:43:04.167237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33233
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33233
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34483/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 14:43:04.284446 5742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:04.284731 5742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:04.284781 5742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:04.288941 5742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:43:04.289006 5742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:04.289029 5742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:04.289047 5742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:43:04.289062 5742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:43:04.289079 5742 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:43:04.289093 5742 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34483/jwks.json
I20260619 14:43:04.293932 5742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33233
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34483/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:04.295184 5742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:04.296577 5742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:04.305279 5747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:04.305287 5748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:04.305404 5742 server_base.cc:1034] running on GCE node
W20260619 14:43:04.305553 5750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:04.306100 5742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:04.307258 5742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:04.308435 5742 hybrid_clock.cc:648] HybridClock initialized: now 1781880184308407 us; error 42 us; skew 500 ppm
I20260619 14:43:04.310339 5742 webserver.cc:460] Webserver started at http://127.0.98.62:35939/ using document root <none> and password file <none>
I20260619 14:43:04.310969 5742 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:04.311059 5742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:04.311282 5742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:04.313448 5742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "cc1558e29c9e4f03b1bae3623e8e8819"
format_stamp: "Formatted at 2026-06-19 14:43:04 on dist-test-slave-cr22"
server_key: "243b1faa0b8502818f5d9b972fe08fbb"
server_key_iv: "78735f1847d502d0567ca0981e88ea12"
server_key_version: "encryptionkey@0"
I20260619 14:43:04.314014 5742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "cc1558e29c9e4f03b1bae3623e8e8819"
format_stamp: "Formatted at 2026-06-19 14:43:04 on dist-test-slave-cr22"
server_key: "243b1faa0b8502818f5d9b972fe08fbb"
server_key_iv: "78735f1847d502d0567ca0981e88ea12"
server_key_version: "encryptionkey@0"
I20260619 14:43:04.317930 5742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 14:43:04.320461 5756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:04.321902 5742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:43:04.322062 5742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "cc1558e29c9e4f03b1bae3623e8e8819"
format_stamp: "Formatted at 2026-06-19 14:43:04 on dist-test-slave-cr22"
server_key: "243b1faa0b8502818f5d9b972fe08fbb"
server_key_iv: "78735f1847d502d0567ca0981e88ea12"
server_key_version: "encryptionkey@0"
I20260619 14:43:04.322181 5742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:04.344816 5742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:04.345571 5742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:04.345786 5742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:04.353264 5742 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33233
I20260619 14:43:04.353278 5808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33233 every 8 connection(s)
I20260619 14:43:04.354264 5742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:43:04.357491 5809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:04.359781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 5742
I20260619 14:43:04.359911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:43:04.360190 392 external_mini_cluster.cc:1428] Setting key 0e11358021af28aba577b1bd05caa591
I20260619 14:43:04.364078 5809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819: Bootstrap starting.
I20260619 14:43:04.366598 5809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:04.367496 5809 log.cc:824] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:04.369611 5809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819: No bootstrap required, opened a new log
I20260619 14:43:04.370420 5812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:04.361529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57714 (local address 127.0.98.62:33233)
0619 14:43:04.362204 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:04.362225 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:04.362267 (+ 42us) server_negotiation.cc:407] Connection header received
0619 14:43:04.363073 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:04.363097 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:04.363618 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:04.363956 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:04.364413 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:04.365852 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:04.366693 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:04.367391 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:04.367587 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:04.368457 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:04.368463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:04.368478 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:04.368949 (+ 471us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:04.369017 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:04.369044 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:04.369143 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:04.369247 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 14:43:04.369507 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":108,"threads_started":1}
I20260619 14:43:04.373287 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:43:04.373495 5809 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc1558e29c9e4f03b1bae3623e8e8819" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33233 } }
I20260619 14:43:04.373715 5809 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:04.373793 5809 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc1558e29c9e4f03b1bae3623e8e8819, State: Initialized, Role: FOLLOWER
I20260619 14:43:04.374282 5809 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc1558e29c9e4f03b1bae3623e8e8819" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33233 } }
I20260619 14:43:04.374431 5809 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:43:04.374495 5809 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:43:04.374596 5809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:04.458873 5809 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc1558e29c9e4f03b1bae3623e8e8819" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33233 } }
I20260619 14:43:04.459468 5809 leader_election.cc:302] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc1558e29c9e4f03b1bae3623e8e8819; no voters:
I20260619 14:43:04.459867 5809 leader_election.cc:288] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:43:04.460017 5814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:04.460266 5814 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819 [term 1 LEADER]: Becoming Leader. State: Replica: cc1558e29c9e4f03b1bae3623e8e8819, State: Running, Role: LEADER
I20260619 14:43:04.460644 5814 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc1558e29c9e4f03b1bae3623e8e8819" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33233 } }
I20260619 14:43:04.461385 5809 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:43:04.461990 5815 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc1558e29c9e4f03b1bae3623e8e8819" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc1558e29c9e4f03b1bae3623e8e8819" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33233 } } }
I20260619 14:43:04.462026 5816 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc1558e29c9e4f03b1bae3623e8e8819. Latest consensus state: current_term: 1 leader_uuid: "cc1558e29c9e4f03b1bae3623e8e8819" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc1558e29c9e4f03b1bae3623e8e8819" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33233 } } }
I20260619 14:43:04.462113 5815 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:04.462141 5816 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:04.462535 5819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:43:04.465902 5819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:43:04.471876 5819 catalog_manager.cc:1347] Generated new cluster ID: 3a5eef3ddd354edaaed98d3517b8c54d
I20260619 14:43:04.471958 5819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:43:04.483330 5819 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:43:04.484684 5819 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:43:04.499032 5819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc1558e29c9e4f03b1bae3623e8e8819: Generated new TSK 0
I20260619 14:43:04.499993 5819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:43:04.521212 5812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:04.515526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57728 (local address 127.0.98.62:33233)
0619 14:43:04.515806 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:04.515811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:04.515909 (+ 98us) server_negotiation.cc:407] Connection header received
0619 14:43:04.516038 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:04.516040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:04.516090 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:04.516188 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:43:04.516434 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:04.517108 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:04.518361 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:04.518604 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:04.518702 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:04.519080 (+ 378us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:43:04.520826 (+ 1746us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 14:43:04.521066 (+ 240us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57728: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":144}
W20260619 14:43:04.521306 5812 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57728: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 14:43:04.523413 5838 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:04.515454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33233 (local address 127.0.0.1:57728)
0619 14:43:04.515781 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:04.515797 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:04.515919 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:04.516207 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:04.516210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:04.516218 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:43:04.516330 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:04.516336 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:04.517217 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:04.517221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:04.518224 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:04.518233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:04.518703 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:04.518728 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:04.518823 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:04.518852 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:43:04.520998 (+ 2146us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 14:43:04.523234 (+ 2236us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33233: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":1944,"thread_start_us":120,"threads_started":1}
I20260619 14:43:04.524071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 5742
E20260619 14:43:04.533666 5839 webserver.cc:533] Webserver: sslize failed:
E20260619 14:43:04.533867 5839 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T14:43:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (407 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T14:43:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:43:04Z Disabled control of system clock
I20260619 14:43:04.573590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41355
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41355
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880155260304-392-0/testchainca.cert with env {}
W20260619 14:43:04.694916 5843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:04.695220 5843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:04.695262 5843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:04.701166 5843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:43:04.701249 5843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:04.701265 5843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:04.701284 5843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:43:04.701300 5843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:43:04.701318 5843 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:43:04.701334 5843 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 14:43:04.707427 5843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41355
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880155260304-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:04.708552 5843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:04.709590 5843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:04.718520 5849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:04.718639 5851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:04.718652 5848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:04.719082 5843 server_base.cc:1034] running on GCE node
I20260619 14:43:04.719980 5843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:04.721247 5843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:04.722450 5843 hybrid_clock.cc:648] HybridClock initialized: now 1781880184722437 us; error 32 us; skew 500 ppm
I20260619 14:43:04.724568 5843 webserver.cc:460] Webserver started at http://127.0.98.62:37995/ using document root <none> and password file <none>
I20260619 14:43:04.725203 5843 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:04.725296 5843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:04.725523 5843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:04.727661 5843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "3b504d0a3cc641b6ac5b2b656c17b622"
format_stamp: "Formatted at 2026-06-19 14:43:04 on dist-test-slave-cr22"
server_key: "417bfb47196b0fdffc932ee1c4e32947"
server_key_iv: "c6a4efd6b4a3cefd0ae970f0f505ecf6"
server_key_version: "encryptionkey@0"
I20260619 14:43:04.728250 5843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b504d0a3cc641b6ac5b2b656c17b622"
format_stamp: "Formatted at 2026-06-19 14:43:04 on dist-test-slave-cr22"
server_key: "417bfb47196b0fdffc932ee1c4e32947"
server_key_iv: "c6a4efd6b4a3cefd0ae970f0f505ecf6"
server_key_version: "encryptionkey@0"
I20260619 14:43:04.732213 5843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:43:04.734897 5857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:04.736470 5843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:43:04.736660 5843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "3b504d0a3cc641b6ac5b2b656c17b622"
format_stamp: "Formatted at 2026-06-19 14:43:04 on dist-test-slave-cr22"
server_key: "417bfb47196b0fdffc932ee1c4e32947"
server_key_iv: "c6a4efd6b4a3cefd0ae970f0f505ecf6"
server_key_version: "encryptionkey@0"
I20260619 14:43:04.736799 5843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:04.766160 5843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:04.766902 5843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:04.767112 5843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:04.774652 5843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41355
I20260619 14:43:04.774897 5909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41355 every 8 connection(s)
I20260619 14:43:04.775734 5843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:43:04.780658 5910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:04.781004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 5843
I20260619 14:43:04.781147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:43:04.781446 392 external_mini_cluster.cc:1428] Setting key 6b51d16d334125f5d6b904cbeec9036d
I20260619 14:43:04.787743 5910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622: Bootstrap starting.
I20260619 14:43:04.790205 5910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:04.791124 5910 log.cc:824] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:04.791697 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:04.782942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39964 (local address 127.0.98.62:41355)
0619 14:43:04.783707 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:04.783724 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:04.783761 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:43:04.784445 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:04.784462 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:04.784918 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:04.785270 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:04.785694 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:04.786655 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:04.787577 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:04.788115 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:04.788287 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:04.789299 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:04.789304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:04.789320 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:04.789730 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:04.789941 (+ 211us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:04.789976 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:04.790280 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:04.790429 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 14:43:04.790733 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":158,"threads_started":1}
I20260619 14:43:04.794154 5910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622: No bootstrap required, opened a new log
I20260619 14:43:04.795876 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:43:04.798835 5910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b504d0a3cc641b6ac5b2b656c17b622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41355 } }
I20260619 14:43:04.799052 5910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:04.799132 5910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b504d0a3cc641b6ac5b2b656c17b622, State: Initialized, Role: FOLLOWER
I20260619 14:43:04.799592 5910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b504d0a3cc641b6ac5b2b656c17b622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41355 } }
I20260619 14:43:04.799734 5910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:43:04.799814 5910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:43:04.799911 5910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:04.857373 5910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b504d0a3cc641b6ac5b2b656c17b622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41355 } }
I20260619 14:43:04.857918 5910 leader_election.cc:302] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b504d0a3cc641b6ac5b2b656c17b622; no voters:
I20260619 14:43:04.858340 5910 leader_election.cc:288] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:43:04.858623 5915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:04.858814 5915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622 [term 1 LEADER]: Becoming Leader. State: Replica: 3b504d0a3cc641b6ac5b2b656c17b622, State: Running, Role: LEADER
I20260619 14:43:04.859141 5915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b504d0a3cc641b6ac5b2b656c17b622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41355 } }
I20260619 14:43:04.859881 5910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:43:04.861058 5917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b504d0a3cc641b6ac5b2b656c17b622. Latest consensus state: current_term: 1 leader_uuid: "3b504d0a3cc641b6ac5b2b656c17b622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b504d0a3cc641b6ac5b2b656c17b622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41355 } } }
I20260619 14:43:04.861094 5916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b504d0a3cc641b6ac5b2b656c17b622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b504d0a3cc641b6ac5b2b656c17b622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41355 } } }
I20260619 14:43:04.861276 5917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:04.861292 5916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:04.861883 5924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:43:04.865186 5924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:43:04.871080 5924 catalog_manager.cc:1347] Generated new cluster ID: aacbe7e879fe4da7babee9208151b14b
I20260619 14:43:04.871161 5924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:43:04.886791 5924 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:43:04.888222 5924 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:43:04.893523 5924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b504d0a3cc641b6ac5b2b656c17b622: Generated new TSK 0
I20260619 14:43:04.894271 5924 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:43:04.945050 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:04.915424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39976 (local address 127.0.98.62:41355)
0619 14:43:04.915577 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:04.915580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:04.915691 (+ 111us) server_negotiation.cc:407] Connection header received
0619 14:43:04.915847 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:04.915849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:04.915895 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:04.916017 (+ 122us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:43:04.916272 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:04.916888 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:04.918176 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:04.918412 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:04.918507 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:04.918889 (+ 382us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:43:04.944641 (+ 25752us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 14:43:04.944931 (+ 290us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39976: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 14:43:04.945145 5913 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39976: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 14:43:04.945304 5939 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:04.915260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.0.1:39976)
0619 14:43:04.915600 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:04.915615 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:04.915730 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:04.916020 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:04.916023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:04.916031 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:43:04.916139 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:04.916153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:04.917000 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:04.917003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:04.917995 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:04.918006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:04.918530 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:04.918562 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:04.918654 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:04.918683 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:43:04.944908 (+ 26225us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 14:43:04.945104 (+ 196us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41355: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":136,"threads_started":1}
I20260619 14:43:04.946048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 5843
2026-06-19T14:43:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (400 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (127 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (124 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:43:05 dist-test-slave-cr22 krb5kdc[5951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:43:05 dist-test-slave-cr22 krb5kdc[5951](info): set up 2 sockets
Jun 19 14:43:05 dist-test-slave-cr22 krb5kdc[5951](info): commencing operation
krb5kdc: starting...
W20260619 14:43:06.252909 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:43:06 dist-test-slave-cr22 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:43:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:43:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:43:06.378158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43211
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:06.487112 5964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:06.487383 5964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:06.487426 5964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:06.491501 5964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:43:06.491562 5964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:06.491582 5964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:06.491600 5964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:43:06.491614 5964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:43:06.497033 5964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:06.498044 5964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:06.499109 5964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:06.507608 5970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:06.507603 5969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:06.507596 5972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:06.507794 5964 server_base.cc:1034] running on GCE node
I20260619 14:43:06.508467 5964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:06.509543 5964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:06.510700 5964 hybrid_clock.cc:648] HybridClock initialized: now 1781880186510683 us; error 31 us; skew 500 ppm
Jun 19 14:43:06 dist-test-slave-cr22 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:06.519940 5964 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:43:06.521052 5964 webserver.cc:460] Webserver started at http://127.0.98.62:41095/ using document root <none> and password file <none>
I20260619 14:43:06.521706 5964 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:06.521802 5964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:06.522049 5964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:06.524657 5964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "347e780777aa4a9d86b2cb1a34439aa5"
format_stamp: "Formatted at 2026-06-19 14:43:06 on dist-test-slave-cr22"
server_key: "709b8ecf9872dbda0882bb189dde16d0"
server_key_iv: "3b7d802a92f5bc9f7e3433c07a1d8e5b"
server_key_version: "encryptionkey@0"
I20260619 14:43:06.525413 5964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "347e780777aa4a9d86b2cb1a34439aa5"
format_stamp: "Formatted at 2026-06-19 14:43:06 on dist-test-slave-cr22"
server_key: "709b8ecf9872dbda0882bb189dde16d0"
server_key_iv: "3b7d802a92f5bc9f7e3433c07a1d8e5b"
server_key_version: "encryptionkey@0"
I20260619 14:43:06.531281 5964 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 14:43:06.535102 5979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:06.536993 5964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:43:06.537199 5964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "347e780777aa4a9d86b2cb1a34439aa5"
format_stamp: "Formatted at 2026-06-19 14:43:06 on dist-test-slave-cr22"
server_key: "709b8ecf9872dbda0882bb189dde16d0"
server_key_iv: "3b7d802a92f5bc9f7e3433c07a1d8e5b"
server_key_version: "encryptionkey@0"
I20260619 14:43:06.537384 5964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:06.568009 5964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:06.572078 5964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:06.572292 5964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:06.580380 5964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43211
I20260619 14:43:06.580678 6031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43211 every 8 connection(s)
I20260619 14:43:06.581403 5964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:43:06.584287 6032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:06.584718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 5964
I20260619 14:43:06.584846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:43:06.585151 392 external_mini_cluster.cc:1428] Setting key 5ab1a4e5b258f1f022a89132b7f43cfa
I20260619 14:43:06.591946 6032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5: Bootstrap starting.
I20260619 14:43:06.594432 6032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5: Neither blocks nor log segments found. Creating new log.
Jun 19 14:43:06 dist-test-slave-cr22 krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:43:06.595266 6032 log.cc:824] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:06.597334 6032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5: No bootstrap required, opened a new log
I20260619 14:43:06.600553 6035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:06.586725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38480 (local address 127.0.98.62:43211)
0619 14:43:06.587176 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:06.587190 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:06.587226 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:43:06.588620 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:06.588640 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:06.588998 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:06.589373 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:06.590268 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:06.591311 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:06.592335 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:06.592843 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:06.593028 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:06.595782 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:06.595788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:06.595801 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:06.595832 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:06.597599 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:06.598068 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:06.598074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:06.598085 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:06.598200 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:06.598505 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:06.598510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:06.598512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:06.598917 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:06.599130 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:06.599466 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 14:43:06.599764 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":130,"threads_started":1}
I20260619 14:43:06.600988 6032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347e780777aa4a9d86b2cb1a34439aa5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43211 } }
I20260619 14:43:06.601199 6032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:06.601253 6032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 347e780777aa4a9d86b2cb1a34439aa5, State: Initialized, Role: FOLLOWER
I20260619 14:43:06.601677 6032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347e780777aa4a9d86b2cb1a34439aa5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43211 } }
I20260619 14:43:06.601806 6032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:43:06.601903 6032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:43:06.602006 6032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:06.602936 6032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347e780777aa4a9d86b2cb1a34439aa5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43211 } }
I20260619 14:43:06.603284 6032 leader_election.cc:302] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 347e780777aa4a9d86b2cb1a34439aa5; no voters:
I20260619 14:43:06.603606 6032 leader_election.cc:288] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:43:06.603720 6037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:06.603984 6037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5 [term 1 LEADER]: Becoming Leader. State: Replica: 347e780777aa4a9d86b2cb1a34439aa5, State: Running, Role: LEADER
I20260619 14:43:06.604305 6037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347e780777aa4a9d86b2cb1a34439aa5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43211 } }
I20260619 14:43:06.604923 6032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:43:06.606154 6039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 347e780777aa4a9d86b2cb1a34439aa5. Latest consensus state: current_term: 1 leader_uuid: "347e780777aa4a9d86b2cb1a34439aa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347e780777aa4a9d86b2cb1a34439aa5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43211 } } }
I20260619 14:43:06.606185 6038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "347e780777aa4a9d86b2cb1a34439aa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347e780777aa4a9d86b2cb1a34439aa5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43211 } } }
I20260619 14:43:06.606314 6039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:06.606317 6038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:06.606665 6046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:43:06.609498 6046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:43:06.619680 6046 catalog_manager.cc:1347] Generated new cluster ID: d224859ccbe44bbcbf859838d21f9c59
I20260619 14:43:06.619776 6046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:43:06.637153 6046 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:43:06.638176 6046 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:43:06.645536 6046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 347e780777aa4a9d86b2cb1a34439aa5: Generated new TSK 0
I20260619 14:43:06.646239 6046 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:43:06.683092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43211
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:06.806011 6058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:06.806303 6058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:06.806377 6058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:06.810576 6058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:06.810669 6058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:06.810755 6058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:43:06.816151 6058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:06.817334 6058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:06.818357 6058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:06.828265 6063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:06.828318 6064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:06.828379 6058 server_base.cc:1034] running on GCE node
W20260619 14:43:06.828320 6066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:06.829041 6058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:06.829829 6058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:06.831035 6058 hybrid_clock.cc:648] HybridClock initialized: now 1781880186831014 us; error 32 us; skew 500 ppm
Jun 19 14:43:06 dist-test-slave-cr22 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:06.833953 6058 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:43:06.834856 6058 webserver.cc:460] Webserver started at http://127.0.98.1:43115/ using document root <none> and password file <none>
I20260619 14:43:06.835408 6058 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:06.835521 6058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:06.835743 6058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:06.837792 6058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "ee39df003965467e99c4cd938a46717d"
format_stamp: "Formatted at 2026-06-19 14:43:06 on dist-test-slave-cr22"
server_key: "3e58d5435070031d94ed95fb3e06d637"
server_key_iv: "0997b9b8c2f908bdcb272a0daccb4360"
server_key_version: "encryptionkey@0"
I20260619 14:43:06.838342 6058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ee39df003965467e99c4cd938a46717d"
format_stamp: "Formatted at 2026-06-19 14:43:06 on dist-test-slave-cr22"
server_key: "3e58d5435070031d94ed95fb3e06d637"
server_key_iv: "0997b9b8c2f908bdcb272a0daccb4360"
server_key_version: "encryptionkey@0"
I20260619 14:43:06.842185 6058 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:43:06.844902 6073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:06.846151 6058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:43:06.846256 6058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "ee39df003965467e99c4cd938a46717d"
format_stamp: "Formatted at 2026-06-19 14:43:06 on dist-test-slave-cr22"
server_key: "3e58d5435070031d94ed95fb3e06d637"
server_key_iv: "0997b9b8c2f908bdcb272a0daccb4360"
server_key_version: "encryptionkey@0"
I20260619 14:43:06.846400 6058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:06.857703 6058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:06.861081 6058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:06.861294 6058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:06.861901 6058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:06.862730 6058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:06.862776 6058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:06.862850 6058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:06.862883 6058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:06.872998 6058 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41595
I20260619 14:43:06.873020 6186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41595 every 8 connection(s)
I20260619 14:43:06.874004 6058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:43:06.880324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 6058
I20260619 14:43:06.880462 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:43:06.880756 392 external_mini_cluster.cc:1428] Setting key 1472ff697a5a2937bec7bfd1142cfc1d
Jun 19 14:43:06 dist-test-slave-cr22 krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:43:06.890028 6035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:06.875961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46555 (local address 127.0.98.62:43211)
0619 14:43:06.876129 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:06.876132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:06.876935 (+ 803us) server_negotiation.cc:407] Connection header received
0619 14:43:06.878447 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:06.878451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:06.878532 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:06.878638 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:06.879927 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:06.880831 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:06.882265 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:06.882615 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:06.882740 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:06.885491 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:06.885499 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:06.885506 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:06.885543 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:06.887491 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:06.888059 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:06.888067 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:06.888073 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:06.888188 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:06.888659 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:06.888667 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:06.888671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:06.888975 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:06.889100 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:06.889631 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 14:43:06.889792 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:43:06.890811 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:06.876280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43211 (local address 127.0.98.1:46555)
0619 14:43:06.876785 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:06.876832 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:06.878229 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:06.878822 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:06.878829 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:06.879196 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:06.879771 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:06.879786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:06.880965 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:06.880969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:06.882116 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:06.882128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:06.882939 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:06.882942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:06.883087 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:06.883768 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:06.883823 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:06.885300 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:06.887614 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:06.887622 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:06.887639 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:06.887935 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:06.888324 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:06.888327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:06.888329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:06.888538 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:06.889119 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:06.889126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:06.889371 (+ 245us) client_negotiation.cc:769] Sending connection context
0619 14:43:06.889547 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 14:43:06.889884 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:43:06.892145 6187 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43211
I20260619 14:43:06.892411 6187 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:06.893055 6187 heartbeater.cc:511] Master 127.0.98.62:43211 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:43:06.894816 5996 ts_manager.cc:194] Registered new tserver with Master: ee39df003965467e99c4cd938a46717d (127.0.98.1:41595)
I20260619 14:43:06.896013 5996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:43:06.912269 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43211
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:07.027266 6192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:07.027628 6192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:07.027717 6192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:07.031898 6192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:07.031955 6192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:07.032018 6192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:43:07.037425 6192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:07.038519 6192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:07.039564 6192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:07.050189 6198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:07.050266 6197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:07.050335 6200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:07.050278 6192 server_base.cc:1034] running on GCE node
I20260619 14:43:07.050906 6192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:07.051719 6192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:07.052911 6192 hybrid_clock.cc:648] HybridClock initialized: now 1781880187052884 us; error 36 us; skew 500 ppm
Jun 19 14:43:07 dist-test-slave-cr22 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:07.055737 6192 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:43:07.056550 6192 webserver.cc:460] Webserver started at http://127.0.98.2:37555/ using document root <none> and password file <none>
I20260619 14:43:07.057088 6192 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:07.057152 6192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:07.057340 6192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:07.059305 6192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "ac73be869db44a9084a28dca45e1f9ad"
format_stamp: "Formatted at 2026-06-19 14:43:07 on dist-test-slave-cr22"
server_key: "a16683b177403a1d662546af41415df9"
server_key_iv: "41e1a6154299f00699a56f42da59be43"
server_key_version: "encryptionkey@0"
I20260619 14:43:07.059881 6192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ac73be869db44a9084a28dca45e1f9ad"
format_stamp: "Formatted at 2026-06-19 14:43:07 on dist-test-slave-cr22"
server_key: "a16683b177403a1d662546af41415df9"
server_key_iv: "41e1a6154299f00699a56f42da59be43"
server_key_version: "encryptionkey@0"
I20260619 14:43:07.063606 6192 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:43:07.066373 6207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:07.067821 6192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:43:07.067958 6192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "ac73be869db44a9084a28dca45e1f9ad"
format_stamp: "Formatted at 2026-06-19 14:43:07 on dist-test-slave-cr22"
server_key: "a16683b177403a1d662546af41415df9"
server_key_iv: "41e1a6154299f00699a56f42da59be43"
server_key_version: "encryptionkey@0"
I20260619 14:43:07.068074 6192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:07.101589 6192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:07.105515 6192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:07.105753 6192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:07.106391 6192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:07.107278 6192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:07.107331 6192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:07.107414 6192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:07.107465 6192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:07.116159 6192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46111
I20260619 14:43:07.116256 6320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46111 every 8 connection(s)
I20260619 14:43:07.117071 6192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:43:07.119248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 6192
I20260619 14:43:07.119388 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:43:07.119820 392 external_mini_cluster.cc:1428] Setting key 8b4ca99b5d6a10374c0f6c856b6b77d3
Jun 19 14:43:07 dist-test-slave-cr22 krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:43:07.134158 6035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:07.118938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40623 (local address 127.0.98.62:43211)
0619 14:43:07.119083 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:07.119088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:07.120113 (+ 1025us) server_negotiation.cc:407] Connection header received
0619 14:43:07.121922 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:07.121925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:07.121969 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:07.122045 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:07.123784 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:07.124657 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:07.126021 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:07.126443 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:07.126578 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:07.129605 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:07.129609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:07.129612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:07.129649 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:07.131393 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:07.132060 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:07.132063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:07.132064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:07.132156 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:07.132578 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:07.132584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:07.132590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:07.132970 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:07.133083 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:07.133909 (+ 826us) server_negotiation.cc:299] Negotiation successful
0619 14:43:07.134022 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:43:07.135128 6323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:07.119333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43211 (local address 127.0.98.2:40623)
0619 14:43:07.119936 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:07.119996 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:07.121677 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:07.122252 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:07.122265 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:07.122829 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:07.123564 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:07.123580 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:07.124824 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:07.124827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:07.125885 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:07.125895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:07.126691 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:07.126695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:07.126845 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:07.127587 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:07.127604 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:07.129410 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:07.131549 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:07.131554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:07.131567 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:07.131934 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:07.132265 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:07.132275 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:07.132276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:07.132416 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:07.133122 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:07.133131 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:07.133508 (+ 377us) client_negotiation.cc:769] Sending connection context
0619 14:43:07.133828 (+ 320us) client_negotiation.cc:240] Negotiation successful
0619 14:43:07.134209 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":127,"threads_started":1}
I20260619 14:43:07.136514 6321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43211
I20260619 14:43:07.136782 6321 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:07.137305 6321 heartbeater.cc:511] Master 127.0.98.62:43211 requested a full tablet report, sending...
I20260619 14:43:07.138360 5996 ts_manager.cc:194] Registered new tserver with Master: ac73be869db44a9084a28dca45e1f9ad (127.0.98.2:46111)
I20260619 14:43:07.138944 5996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:43:07.150671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43211
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:07.268033 6326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:07.268338 6326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:07.268414 6326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:07.273079 6326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:07.273177 6326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:07.273272 6326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:43:07.279214 6326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:07.280373 6326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:07.281441 6326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:07.291425 6332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:07.291466 6334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:07.291615 6326 server_base.cc:1034] running on GCE node
W20260619 14:43:07.291491 6331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:07.292223 6326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:07.292870 6326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:07.294057 6326 hybrid_clock.cc:648] HybridClock initialized: now 1781880187294075 us; error 72 us; skew 500 ppm
Jun 19 14:43:07 dist-test-slave-cr22 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:07.296873 6326 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:43:07.297768 6326 webserver.cc:460] Webserver started at http://127.0.98.3:43159/ using document root <none> and password file <none>
I20260619 14:43:07.298292 6326 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:07.298357 6326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:07.298524 6326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:07.300661 6326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "4baa574a72a24bd9812cf8e4fdc52155"
format_stamp: "Formatted at 2026-06-19 14:43:07 on dist-test-slave-cr22"
server_key: "a420b67f62c995044fc0ffcfb5dde0c3"
server_key_iv: "1a4fc4ad3a4e86946d407a1e061b0bab"
server_key_version: "encryptionkey@0"
I20260619 14:43:07.301160 6326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4baa574a72a24bd9812cf8e4fdc52155"
format_stamp: "Formatted at 2026-06-19 14:43:07 on dist-test-slave-cr22"
server_key: "a420b67f62c995044fc0ffcfb5dde0c3"
server_key_iv: "1a4fc4ad3a4e86946d407a1e061b0bab"
server_key_version: "encryptionkey@0"
I20260619 14:43:07.305210 6326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:43:07.307809 6341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:07.309103 6326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:43:07.309199 6326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "4baa574a72a24bd9812cf8e4fdc52155"
format_stamp: "Formatted at 2026-06-19 14:43:07 on dist-test-slave-cr22"
server_key: "a420b67f62c995044fc0ffcfb5dde0c3"
server_key_iv: "1a4fc4ad3a4e86946d407a1e061b0bab"
server_key_version: "encryptionkey@0"
I20260619 14:43:07.309283 6326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:07.321041 6326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:07.324082 6326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:07.324235 6326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:07.324788 6326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:07.325562 6326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:07.325608 6326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:07.325646 6326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:07.325667 6326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:07.334334 6326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34945
I20260619 14:43:07.334385 6454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34945 every 8 connection(s)
I20260619 14:43:07.335175 6326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:43:07.337273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 6326
I20260619 14:43:07.337400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:43:07.337687 392 external_mini_cluster.cc:1428] Setting key 8e0a9c5548e3bf2e65ead5e59ff7cae9
Jun 19 14:43:07 dist-test-slave-cr22 krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:43:07.348417 6035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:07.336988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53631 (local address 127.0.98.62:43211)
0619 14:43:07.337158 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:07.337164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:07.338014 (+ 850us) server_negotiation.cc:407] Connection header received
0619 14:43:07.339345 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:07.339348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:07.339402 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:07.339513 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:07.340619 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:07.341151 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:07.342275 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:07.342520 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:07.342627 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:07.345138 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:07.345141 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:07.345143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:07.345165 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:07.346421 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:07.346966 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:07.346969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:07.346970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:07.347041 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:07.347481 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:07.347486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:07.347488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:07.347664 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:07.347745 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:07.348194 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 14:43:07.348291 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:43:07.349268 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:07.337339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43211 (local address 127.0.98.3:53631)
0619 14:43:07.337841 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:07.337892 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:07.339156 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:07.339686 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:07.339693 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:07.340033 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:07.340483 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:07.340495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:07.341272 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:07.341275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:07.342145 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:07.342154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:07.342659 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:07.342663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:07.342838 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:07.343324 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:07.343340 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:07.344990 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:07.346537 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:07.346541 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:07.346552 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:07.346882 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:07.347160 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:07.347163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:07.347165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:07.347340 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:07.347764 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:07.347769 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:07.347987 (+ 218us) client_negotiation.cc:769] Sending connection context
0619 14:43:07.348150 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 14:43:07.348406 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":129,"threads_started":1}
I20260619 14:43:07.350579 6455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43211
I20260619 14:43:07.350847 6455 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:07.351419 6455 heartbeater.cc:511] Master 127.0.98.62:43211 requested a full tablet report, sending...
I20260619 14:43:07.352454 5996 ts_manager.cc:194] Registered new tserver with Master: 4baa574a72a24bd9812cf8e4fdc52155 (127.0.98.3:34945)
I20260619 14:43:07.353004 5996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53631
I20260619 14:43:07.363219 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/test-admin.keytab.
Jun 19 14:43:07 dist-test-slave-cr22 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:07.379899 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 14:43:07.382676 6462 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:07.381205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43082 (local address 127.0.98.1:41595)
0619 14:43:07.381482 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:07.381499 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:07.381528 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:43:07.381596 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:07.381601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:07.381798 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:07.382010 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:07.382485 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43082: BlockingRecv error: recv got EOF from 127.0.0.1:43082 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
W20260619 14:43:07.382903 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/krb5cc)
Jun 19 14:43:07 dist-test-slave-cr22 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:07.383930 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:43:07 dist-test-slave-cr22 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:07.384814 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:43:07 dist-test-slave-cr22 krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:43:07.394289 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:07.385606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.0.98.1:41595)
0619 14:43:07.385747 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:07.385751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:07.385765 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:43:07.385896 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:07.385898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:07.385937 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:07.386006 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:07.386497 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:07.387278 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:07.388133 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:07.388500 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:07.388583 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:07.390693 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:07.390701 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:07.390713 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:07.390736 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:07.392489 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:07.392989 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:07.392995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:07.392996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:07.393081 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:07.393307 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:07.393310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:07.393319 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:07.393556 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:07.393716 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:07.393947 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 14:43:07.394118 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 14:43:07.398128 6462 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:07.397188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.0.98.1:41595)
0619 14:43:07.397325 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:07.397328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:07.397341 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:43:07.397488 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:07.397490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:07.397529 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:07.397603 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:07.398013 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43098: BlockingRecv error: recv got EOF from 127.0.0.1:43098 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 14:43:07.398216 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 14:43:07 dist-test-slave-cr22 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:07.399196 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:43:07 dist-test-slave-cr22 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:07.400054 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:43:07 dist-test-slave-cr22 krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:43:07.408753 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:07.400812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43106 (local address 127.0.98.1:41595)
0619 14:43:07.400940 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:07.400943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:07.400984 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:43:07.401068 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:07.401070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:07.401102 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:07.401164 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:07.401590 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:07.402083 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:07.402889 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:07.403125 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:07.403206 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:07.405521 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:07.405524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:07.405527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:07.405547 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:07.407160 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:07.407570 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:07.407573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:07.407574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:07.407653 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:07.407922 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:07.407925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:07.407926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:07.408097 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:07.408210 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:07.408440 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 14:43:07.408563 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 14:43:07.411516 6462 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:07.410436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43116 (local address 127.0.98.1:41595)
0619 14:43:07.410555 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:07.410559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:07.410744 (+ 185us) server_negotiation.cc:407] Connection header received
0619 14:43:07.410878 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:07.410882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:07.410924 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:07.411010 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:07.411393 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43116: BlockingRecv error: recv got EOF from 127.0.0.1:43116 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 14:43:07.411798 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880155260304-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 14:43:07 dist-test-slave-cr22 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:07.412833 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:43:07 dist-test-slave-cr22 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:07.413822 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:43:07 dist-test-slave-cr22 krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:43:07.422271 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:07.414611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43120 (local address 127.0.98.1:41595)
0619 14:43:07.414793 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:07.414797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:07.414913 (+ 116us) server_negotiation.cc:407] Connection header received
0619 14:43:07.415030 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:07.415033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:07.415070 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:07.415158 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:07.415687 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:07.416192 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:07.416986 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:07.417233 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:07.417343 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:07.419393 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:07.419397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:07.419400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:07.419429 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:07.420704 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:07.421130 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:07.421135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:07.421137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:07.421218 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:07.421484 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:07.421487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:07.421489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:07.421647 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:07.421777 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:07.422013 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 14:43:07.422119 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260619 14:43:07.423384 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 6058
I20260619 14:43:07.428966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 6192
I20260619 14:43:07.434165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 6326
I20260619 14:43:07.441099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 5964
2026-06-19T14:43:07Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2244 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:43:07 dist-test-slave-cr22 krb5kdc[6475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:43:07 dist-test-slave-cr22 krb5kdc[6475](info): set up 2 sockets
Jun 19 14:43:07 dist-test-slave-cr22 krb5kdc[6475](info): commencing operation
krb5kdc: starting...
W20260619 14:43:08.540326 392 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:43:08 dist-test-slave-cr22 krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:43:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:43:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 14:43:08.671164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35593
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:08.781831 6488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:08.782145 6488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:08.782191 6488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:08.786854 6488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:43:08.786929 6488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:08.786952 6488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:08.786971 6488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:43:08.786989 6488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:43:08.792973 6488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:08.793996 6488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:08.794945 6488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:08.803764 6493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:08.803890 6494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:08.803895 6488 server_base.cc:1034] running on GCE node
W20260619 14:43:08.804090 6496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:08.804648 6488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:08.805742 6488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:08.806993 6488 hybrid_clock.cc:648] HybridClock initialized: now 1781880188806959 us; error 50 us; skew 500 ppm
Jun 19 14:43:08 dist-test-slave-cr22 krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880188, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:08.810026 6488 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 14:43:08.811030 6488 webserver.cc:460] Webserver started at http://127.0.98.62:40727/ using document root <none> and password file <none>
I20260619 14:43:08.811679 6488 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:08.811748 6488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:08.811939 6488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:08.814073 6488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "4f0cad194c5742c692d7152d92cd5565"
format_stamp: "Formatted at 2026-06-19 14:43:08 on dist-test-slave-cr22"
server_key: "1f09be2a76023b3f72194e3ed03e0b5a"
server_key_iv: "bab24c9dd688e784d9e13ee4dc128387"
server_key_version: "encryptionkey@0"
I20260619 14:43:08.814600 6488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "4f0cad194c5742c692d7152d92cd5565"
format_stamp: "Formatted at 2026-06-19 14:43:08 on dist-test-slave-cr22"
server_key: "1f09be2a76023b3f72194e3ed03e0b5a"
server_key_iv: "bab24c9dd688e784d9e13ee4dc128387"
server_key_version: "encryptionkey@0"
I20260619 14:43:08.818560 6488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 14:43:08.821266 6503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:08.822553 6488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:43:08.822764 6488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "4f0cad194c5742c692d7152d92cd5565"
format_stamp: "Formatted at 2026-06-19 14:43:08 on dist-test-slave-cr22"
server_key: "1f09be2a76023b3f72194e3ed03e0b5a"
server_key_iv: "bab24c9dd688e784d9e13ee4dc128387"
server_key_version: "encryptionkey@0"
I20260619 14:43:08.822888 6488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:08.855590 6488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:08.858868 6488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:08.859110 6488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:08.866477 6488 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35593
I20260619 14:43:08.866827 6555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35593 every 8 connection(s)
I20260619 14:43:08.867515 6488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:43:08.867688 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 6488
I20260619 14:43:08.867820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:43:08.868144 392 external_mini_cluster.cc:1428] Setting key 352394005c28111558336414fa142170
I20260619 14:43:08.870718 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:08.876482 6556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565: Bootstrap starting.
Jun 19 14:43:08 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:43:08.879071 6556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:08.879860 6556 log.cc:824] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:08.881816 6556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565: No bootstrap required, opened a new log
I20260619 14:43:08.882592 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:08.869489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42458 (local address 127.0.98.62:35593)
0619 14:43:08.869996 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:08.870010 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:08.870045 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:43:08.870664 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:08.870679 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:08.871061 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:08.871392 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:08.872216 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:08.873298 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:08.874127 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:08.874582 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:08.874753 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:08.877849 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:08.877855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:08.877868 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:08.877895 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:08.879651 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:08.880096 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:08.880100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:08.880104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:08.880242 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:08.880578 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:08.880581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:08.880583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:08.880982 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:08.881181 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:08.881477 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 14:43:08.881822 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":152,"threads_started":1}
I20260619 14:43:08.885751 6556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } }
I20260619 14:43:08.886001 6556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:08.886085 6556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f0cad194c5742c692d7152d92cd5565, State: Initialized, Role: FOLLOWER
I20260619 14:43:08.886498 6556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } }
I20260619 14:43:08.886646 6556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:43:08.886726 6556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:43:08.886840 6556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:08.887941 6556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } }
I20260619 14:43:08.888345 6556 leader_election.cc:302] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f0cad194c5742c692d7152d92cd5565; no voters:
I20260619 14:43:08.888679 6556 leader_election.cc:288] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:43:08.888752 6561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:08.888988 6561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [term 1 LEADER]: Becoming Leader. State: Replica: 4f0cad194c5742c692d7152d92cd5565, State: Running, Role: LEADER
I20260619 14:43:08.889317 6561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } }
I20260619 14:43:08.889909 6556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:43:08.890939 6562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f0cad194c5742c692d7152d92cd5565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } }
I20260619 14:43:08.891065 6562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:08.891338 6563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f0cad194c5742c692d7152d92cd5565. Latest consensus state: current_term: 1 leader_uuid: "4f0cad194c5742c692d7152d92cd5565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } }
I20260619 14:43:08.891429 6563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:08.891551 6570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:43:08.894279 6570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:43:08.899885 6570 catalog_manager.cc:1347] Generated new cluster ID: c48d93b0efcf4a02a9e36ec812ca32d4
I20260619 14:43:08.899958 6570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:43:08.920049 6570 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:43:08.920969 6570 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:43:08.927783 6570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565: Generated new TSK 0
I20260619 14:43:08.928488 6570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 14:43:08.959277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35593
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:09.077031 6582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:09.077350 6582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:09.077437 6582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:09.081596 6582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:09.081692 6582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:09.081791 6582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:43:09.087155 6582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:09.088299 6582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:09.089319 6582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:09.100440 6587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:09.100461 6588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:09.100461 6590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:09.100768 6582 server_base.cc:1034] running on GCE node
I20260619 14:43:09.101365 6582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:09.101995 6582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:09.103168 6582 hybrid_clock.cc:648] HybridClock initialized: now 1781880189103140 us; error 40 us; skew 500 ppm
Jun 19 14:43:09 dist-test-slave-cr22 krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880189, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:09.106428 6582 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 14:43:09.107524 6582 webserver.cc:460] Webserver started at http://127.0.98.1:34923/ using document root <none> and password file <none>
I20260619 14:43:09.108103 6582 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:09.108163 6582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:09.108392 6582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:09.110512 6582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "bf2f8ba8faff4d7dab0982c9b75bf143"
format_stamp: "Formatted at 2026-06-19 14:43:09 on dist-test-slave-cr22"
server_key: "f9e6a25529d6bee14757c0a978cb56ac"
server_key_iv: "c77b302c87a670c8c5ea106ef825ad70"
server_key_version: "encryptionkey@0"
I20260619 14:43:09.111064 6582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bf2f8ba8faff4d7dab0982c9b75bf143"
format_stamp: "Formatted at 2026-06-19 14:43:09 on dist-test-slave-cr22"
server_key: "f9e6a25529d6bee14757c0a978cb56ac"
server_key_iv: "c77b302c87a670c8c5ea106ef825ad70"
server_key_version: "encryptionkey@0"
I20260619 14:43:09.114743 6582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:43:09.117106 6597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:09.118309 6582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:43:09.118439 6582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "bf2f8ba8faff4d7dab0982c9b75bf143"
format_stamp: "Formatted at 2026-06-19 14:43:09 on dist-test-slave-cr22"
server_key: "f9e6a25529d6bee14757c0a978cb56ac"
server_key_iv: "c77b302c87a670c8c5ea106ef825ad70"
server_key_version: "encryptionkey@0"
I20260619 14:43:09.118546 6582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:09.129505 6582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:09.132733 6582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:09.132943 6582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:09.133543 6582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:09.134377 6582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:09.134447 6582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:09.134519 6582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:09.134559 6582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:09.143522 6582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38627
I20260619 14:43:09.143540 6710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38627 every 8 connection(s)
I20260619 14:43:09.144387 6582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:43:09.146509 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 6582
I20260619 14:43:09.146637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:43:09.146905 392 external_mini_cluster.cc:1428] Setting key d3cc887f03fc94cb6d7dea8352e17c86
Jun 19 14:43:09 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880189, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260619 14:43:09.160095 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.146285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58981 (local address 127.0.98.62:35593)
0619 14:43:09.146563 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:09.146568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:09.147707 (+ 1139us) server_negotiation.cc:407] Connection header received
0619 14:43:09.149234 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:09.149240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:09.149302 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:09.149395 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:09.150737 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.151337 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.152616 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.152870 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.152988 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:09.156295 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:09.156298 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:09.156300 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:09.156323 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:09.157871 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.158469 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.158472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.158474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.158557 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.158910 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.158913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.158915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.159097 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:09.159212 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:09.159794 (+ 582us) server_negotiation.cc:299] Negotiation successful
0619 14:43:09.159924 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260619 14:43:09.160908 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.146760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.1:58981)
0619 14:43:09.147561 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:09.147614 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:09.149011 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:09.149585 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:09.149595 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:09.150003 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:09.150568 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:09.150583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.151493 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.151497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:09.152458 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:09.152468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.153025 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.153030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:09.153284 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:09.153990 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:09.154020 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:09.156052 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:09.157986 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:09.157992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:09.158003 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:09.158351 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:09.158665 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:09.158674 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:09.158676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:09.158810 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:09.159223 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:09.159232 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:09.159550 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 14:43:09.159759 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 14:43:09.160083 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":123,"threads_started":1}
I20260619 14:43:09.162317 6711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35593
I20260619 14:43:09.162560 6711 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:09.163086 6711 heartbeater.cc:511] Master 127.0.98.62:35593 requested a full tablet report, sending...
I20260619 14:43:09.164710 6518 ts_manager.cc:194] Registered new tserver with Master: bf2f8ba8faff4d7dab0982c9b75bf143 (127.0.98.1:38627)
I20260619 14:43:09.165903 6518 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 14:43:09.178644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35593
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:09.289307 6716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:09.289635 6716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:09.289706 6716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:09.293990 6716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:09.294086 6716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:09.294179 6716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:43:09.299556 6716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:09.300576 6716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:09.301553 6716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:09.311421 6722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:09.311408 6721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:09.311528 6716 server_base.cc:1034] running on GCE node
W20260619 14:43:09.311414 6724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:09.312063 6716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:09.312656 6716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:09.313818 6716 hybrid_clock.cc:648] HybridClock initialized: now 1781880189313792 us; error 40 us; skew 500 ppm
Jun 19 14:43:09 dist-test-slave-cr22 krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880189, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:09.316454 6716 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 14:43:09.317202 6716 webserver.cc:460] Webserver started at http://127.0.98.2:44361/ using document root <none> and password file <none>
I20260619 14:43:09.317682 6716 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:09.317739 6716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:09.317903 6716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:09.319857 6716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "23ad98e2a58d4320aaf58238a845e40a"
format_stamp: "Formatted at 2026-06-19 14:43:09 on dist-test-slave-cr22"
server_key: "a98d7dd7f4f2ba3b34cfc71738cde456"
server_key_iv: "cd679f0cacffd57d45fd376d9093e030"
server_key_version: "encryptionkey@0"
I20260619 14:43:09.320338 6716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "23ad98e2a58d4320aaf58238a845e40a"
format_stamp: "Formatted at 2026-06-19 14:43:09 on dist-test-slave-cr22"
server_key: "a98d7dd7f4f2ba3b34cfc71738cde456"
server_key_iv: "cd679f0cacffd57d45fd376d9093e030"
server_key_version: "encryptionkey@0"
I20260619 14:43:09.323860 6716 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:43:09.326181 6731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:09.327566 6716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:43:09.327685 6716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "23ad98e2a58d4320aaf58238a845e40a"
format_stamp: "Formatted at 2026-06-19 14:43:09 on dist-test-slave-cr22"
server_key: "a98d7dd7f4f2ba3b34cfc71738cde456"
server_key_iv: "cd679f0cacffd57d45fd376d9093e030"
server_key_version: "encryptionkey@0"
I20260619 14:43:09.327800 6716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:09.349867 6716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:09.352772 6716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:09.352968 6716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:09.353621 6716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:09.354423 6716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:09.354489 6716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:09.354558 6716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:09.354598 6716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:09.363366 6716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34115
I20260619 14:43:09.363389 6844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34115 every 8 connection(s)
I20260619 14:43:09.364270 6716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:43:09.365051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 6716
I20260619 14:43:09.365171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:43:09.365456 392 external_mini_cluster.cc:1428] Setting key 83a757fdded890111ee5ed3d12e7ce7c
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Jun 19 14:43:09 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880189, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260619 14:43:09.378597 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.367039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50997 (local address 127.0.98.62:35593)
0619 14:43:09.367166 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:09.367170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:09.368233 (+ 1063us) server_negotiation.cc:407] Connection header received
0619 14:43:09.369534 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:09.369537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:09.369589 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:09.369668 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:09.370763 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.371283 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.372436 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.372662 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.372771 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:09.375111 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:09.375114 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:09.375117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:09.375146 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:09.376610 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.377147 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.377149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.377151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.377221 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.377543 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.377547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.377549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.377721 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:09.377842 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:09.378304 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 14:43:09.378424 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:43:09.379340 6847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.367581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.2:50997)
0619 14:43:09.368088 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:09.368126 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:09.369339 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:09.369834 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:09.369841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:09.370180 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:09.370632 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:09.370643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.371403 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.371406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:09.372312 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:09.372321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.372805 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.372810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:09.372988 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:09.373574 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:09.373590 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:09.374948 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:09.376711 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:09.376716 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:09.376727 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:09.377022 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:09.377313 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:09.377316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:09.377318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:09.377451 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:09.377851 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:09.377859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:09.378080 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 14:43:09.378257 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 14:43:09.378562 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":157,"threads_started":1}
I20260619 14:43:09.380474 6845 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35593
I20260619 14:43:09.380726 6845 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:09.381177 6845 heartbeater.cc:511] Master 127.0.98.62:35593 requested a full tablet report, sending...
I20260619 14:43:09.382225 6518 ts_manager.cc:194] Registered new tserver with Master: 23ad98e2a58d4320aaf58238a845e40a (127.0.98.2:34115)
I20260619 14:43:09.382782 6518 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 14:43:09.393767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35593
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:09.504480 6850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:09.504751 6850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:09.504796 6850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:09.508912 6850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:09.508967 6850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:09.509027 6850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:43:09.514325 6850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:09.515303 6850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:09.516260 6850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:09.526254 6855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:09.526264 6858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:09.526283 6856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:09.526361 6850 server_base.cc:1034] running on GCE node
I20260619 14:43:09.526961 6850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:09.527696 6850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:09.528910 6850 hybrid_clock.cc:648] HybridClock initialized: now 1781880189528878 us; error 46 us; skew 500 ppm
Jun 19 14:43:09 dist-test-slave-cr22 krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880189, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:09.531966 6850 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 14:43:09.533066 6850 webserver.cc:460] Webserver started at http://127.0.98.3:40861/ using document root <none> and password file <none>
I20260619 14:43:09.533646 6850 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:09.533737 6850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:09.533970 6850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:09.536121 6850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "875fc9e2d4544467a24fce99d075ed32"
format_stamp: "Formatted at 2026-06-19 14:43:09 on dist-test-slave-cr22"
server_key: "585ee5c8df625d74e23971051adc5fee"
server_key_iv: "1b4bcb288be9a342f520d5c9f3987bb0"
server_key_version: "encryptionkey@0"
I20260619 14:43:09.536711 6850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "875fc9e2d4544467a24fce99d075ed32"
format_stamp: "Formatted at 2026-06-19 14:43:09 on dist-test-slave-cr22"
server_key: "585ee5c8df625d74e23971051adc5fee"
server_key_iv: "1b4bcb288be9a342f520d5c9f3987bb0"
server_key_version: "encryptionkey@0"
I20260619 14:43:09.540719 6850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 14:43:09.543320 6865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:09.544546 6850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:43:09.544684 6850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "875fc9e2d4544467a24fce99d075ed32"
format_stamp: "Formatted at 2026-06-19 14:43:09 on dist-test-slave-cr22"
server_key: "585ee5c8df625d74e23971051adc5fee"
server_key_iv: "1b4bcb288be9a342f520d5c9f3987bb0"
server_key_version: "encryptionkey@0"
I20260619 14:43:09.544796 6850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:09.568048 6850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:09.571158 6850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:09.571359 6850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:09.571972 6850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:09.572840 6850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:09.572904 6850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:09.572958 6850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:09.572981 6850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:09.582108 6850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33247
I20260619 14:43:09.582125 6978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33247 every 8 connection(s)
I20260619 14:43:09.583019 6850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:43:09.590225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 6850
I20260619 14:43:09.590382 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:43:09.590734 392 external_mini_cluster.cc:1428] Setting key 7274cfe2f548775ec8135b2f30f675c4
Jun 19 14:43:09 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880189, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:43:09.596521 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.584836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47911 (local address 127.0.98.62:35593)
0619 14:43:09.585002 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:09.585007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:09.586261 (+ 1254us) server_negotiation.cc:407] Connection header received
0619 14:43:09.587122 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:09.587125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:09.587172 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:09.587252 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:09.588513 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.589069 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.590244 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.590602 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.590715 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:09.593484 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:09.593486 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:09.593489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:09.593512 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:09.594734 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.595193 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.595196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.595197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.595271 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.595607 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.595609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.595610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.595778 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:09.595889 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:09.596304 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 14:43:09.596403 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:43:09.597287 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.585219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.3:47911)
0619 14:43:09.586107 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:09.586150 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:09.586929 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:09.587524 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:09.587531 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:09.587893 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:09.588375 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:09.588389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.589182 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.589184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:09.590123 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:09.590132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.590744 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.590747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:09.590960 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:09.591709 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:09.591733 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:09.593309 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:09.594848 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:09.594852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:09.594863 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:09.595108 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:09.595347 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:09.595349 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:09.595351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:09.595528 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:09.595885 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:09.595889 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:09.596104 (+ 215us) client_negotiation.cc:769] Sending connection context
0619 14:43:09.596248 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 14:43:09.596486 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":555,"threads_started":1}
I20260619 14:43:09.598343 6979 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35593
I20260619 14:43:09.598558 6979 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:09.599012 6979 heartbeater.cc:511] Master 127.0.98.62:35593 requested a full tablet report, sending...
I20260619 14:43:09.600011 6518 ts_manager.cc:194] Registered new tserver with Master: 875fc9e2d4544467a24fce99d075ed32 (127.0.98.3:33247)
I20260619 14:43:09.600565 6518 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47911
I20260619 14:43:09.604743 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260619 14:43:09.633044 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34175
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35593,127.0.98.61:34175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:09.745599 6984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:09.745873 6984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:09.745913 6984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:09.750025 6984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:43:09.750093 6984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:09.750114 6984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:09.750168 6984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:43:09.750197 6984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:43:09.755483 6984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35593,127.0.98.61:34175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
W20260619 14:43:09.755712 6984 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:35593,127.0.98.61:34175'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 14:43:09.756599 6984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:09.757613 6984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:09.766299 6992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:09.766348 6989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:09.766350 6990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:09.766767 6984 server_base.cc:1034] running on GCE node
I20260619 14:43:09.767316 6984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:09.768491 6984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:09.769668 6984 hybrid_clock.cc:648] HybridClock initialized: now 1781880189769643 us; error 41 us; skew 500 ppm
Jun 19 14:43:09 dist-test-slave-cr22 krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880189, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:09.772621 6984 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 14:43:09.773597 6984 webserver.cc:460] Webserver started at http://127.0.98.61:41783/ using document root <none> and password file <none>
I20260619 14:43:09.774151 6984 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:09.774232 6984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:09.774449 6984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:09.776532 6984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/data/instance:
uuid: "a918948481984a82bef65cbedfad5ef7"
format_stamp: "Formatted at 2026-06-19 14:43:09 on dist-test-slave-cr22"
server_key: "f662bf7ef7f87592bdc2c41644c3eea0"
server_key_iv: "120c73685f0024b642bf989d79e9522a"
server_key_version: "encryptionkey@0"
I20260619 14:43:09.777093 6984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/wal/instance:
uuid: "a918948481984a82bef65cbedfad5ef7"
format_stamp: "Formatted at 2026-06-19 14:43:09 on dist-test-slave-cr22"
server_key: "f662bf7ef7f87592bdc2c41644c3eea0"
server_key_iv: "120c73685f0024b642bf989d79e9522a"
server_key_version: "encryptionkey@0"
I20260619 14:43:09.781184 6984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:43:09.783702 6999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:09.785018 6984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:43:09.785167 6984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/wal
uuid: "a918948481984a82bef65cbedfad5ef7"
format_stamp: "Formatted at 2026-06-19 14:43:09 on dist-test-slave-cr22"
server_key: "f662bf7ef7f87592bdc2c41644c3eea0"
server_key_iv: "120c73685f0024b642bf989d79e9522a"
server_key_version: "encryptionkey@0"
I20260619 14:43:09.785292 6984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:09.797222 6984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:09.800133 6984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:09.800338 6984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:09.807148 6984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34175
I20260619 14:43:09.807422 7051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34175 every 8 connection(s)
I20260619 14:43:09.808175 6984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/data/info.pb
I20260619 14:43:09.809410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 6984
I20260619 14:43:09.811441 7052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:09.815871 7052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:43:09 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 14:43:09 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880189, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:43:09.825878 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.811126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.0.98.61:34175)
0619 14:43:09.811614 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:09.811634 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:09.811680 (+ 46us) server_negotiation.cc:407] Connection header received
0619 14:43:09.812688 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:09.812709 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:09.813086 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:09.813432 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:09.814332 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.815598 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.816707 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.817235 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.817418 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:09.820663 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:09.820668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:09.820677 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:09.820702 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:09.822667 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.823101 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.823109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.823115 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.823273 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.823645 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.823650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.823653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.824162 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:09.824364 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:09.824678 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 14:43:09.824983 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":164,"threads_started":1}
I20260619 14:43:09.828325 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.817580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.0.1:42466)
0619 14:43:09.818301 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:09.818354 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:09.818488 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:09.818785 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:09.818792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:09.819286 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:09.819505 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:09.819513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.820474 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.820478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:09.821195 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:09.821203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.821864 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.821870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:09.822017 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:09.822580 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:09.822599 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:09.824281 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:09.826380 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:09.826391 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:09.826397 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:09.826634 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:09.826968 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:09.826971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:09.826973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:09.827084 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:09.827696 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:09.827702 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:09.827816 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 14:43:09.827960 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 14:43:09.828143 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":133,"threads_started":1}
I20260619 14:43:09.828621 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.817615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.0.98.62:35593)
0619 14:43:09.817784 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:09.817788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:09.818447 (+ 659us) server_negotiation.cc:407] Connection header received
0619 14:43:09.818587 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:09.818589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:09.818632 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:09.818703 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:09.819619 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.820343 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.821296 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.821733 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.821838 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:09.824433 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:09.824437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:09.824440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:09.824472 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:09.826267 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.826752 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.826757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.826759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.826853 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.827219 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.827221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.827223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.827596 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:09.827685 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:09.828325 (+ 640us) server_negotiation.cc:299] Negotiation successful
0619 14:43:09.828475 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:43:09.829619 7052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:43:09 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880189, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 14:43:09.838038 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.830097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34175 (local address 127.0.0.1:48354)
0619 14:43:09.830277 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:09.830292 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:09.830361 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:09.830556 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:09.830558 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:09.830687 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:09.830844 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:09.830850 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.831969 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.831971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:09.832552 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:09.832560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.832973 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.832975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:09.833038 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:09.833434 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:09.833445 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:09.834801 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:09.836655 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:09.836658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:09.836659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:09.836856 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:09.837088 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:09.837098 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:09.837099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:09.837169 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:09.837550 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:09.837553 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:09.837702 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 14:43:09.837805 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 14:43:09.837912 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 14:43:09.838160 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.830201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48354 (local address 127.0.98.61:34175)
0619 14:43:09.830312 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:09.830316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:09.830335 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:43:09.830432 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:09.830434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:09.830465 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:09.830528 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:09.830937 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.831866 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.832644 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.832887 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.832964 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:09.834984 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:09.834988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:09.834990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:09.835009 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:09.836521 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.836940 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.836942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.836943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.837012 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.837266 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.837269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.837270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.837438 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:09.837522 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:09.837904 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 14:43:09.837992 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 14:43:09.840950 7052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: Bootstrap starting.
I20260619 14:43:09.843179 7052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:09.843959 7052 log.cc:824] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:09.845791 7052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: No bootstrap required, opened a new log
I20260619 14:43:09.849041 7052 raft_consensus.cc:348] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } }
I20260619 14:43:09.849277 7052 raft_consensus.cc:374] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:09.849349 7052 raft_consensus.cc:729] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a918948481984a82bef65cbedfad5ef7, State: Initialized, Role: FOLLOWER
I20260619 14:43:09.849753 7052 consensus_queue.cc:260] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } }
I20260619 14:43:09.850235 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } } }
I20260619 14:43:09.850346 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:43:09.850953 7052 sys_catalog.cc:564] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:43:09.855402 7069 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:43:09.855518 7069 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:43:09.857358 6984 master_runner.cc:428] Detected that this master a918948481984a82bef65cbedfad5ef7 is joining an existing cluster
I20260619 14:43:09.857429 6984 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:34175
I20260619 14:43:09.867977 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.859792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.0.98.61:34175)
0619 14:43:09.859946 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:09.859950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:09.860166 (+ 216us) server_negotiation.cc:407] Connection header received
0619 14:43:09.860297 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:09.860302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:09.860340 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:09.860408 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:09.861022 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.861788 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.862996 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.863199 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.863310 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:09.864561 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:09.864568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:09.864573 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:09.864597 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:09.866406 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.866845 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.866848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.866850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.866916 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.867155 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.867157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.867159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.867336 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:09.867402 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:09.867699 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 14:43:09.867823 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:43:09.868369 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.859740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34175 (local address 127.0.0.1:48362)
0619 14:43:09.860075 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:09.860089 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:09.860199 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:09.860445 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:09.860448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:09.860764 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:09.860914 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:09.860922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.861916 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.861922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:09.862869 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:09.862879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.863343 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.863346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:09.863429 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:09.863962 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:09.863976 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:09.864423 (+ 447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:09.866515 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:09.866518 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:09.866520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:09.866742 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:09.867004 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:09.867006 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:09.867008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:09.867075 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:09.867479 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:09.867482 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:09.867555 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 14:43:09.868095 (+ 540us) client_negotiation.cc:240] Negotiation successful
0619 14:43:09.868209 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":10,"thread_start_us":99,"threads_started":1}
I20260619 14:43:09.870038 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.859439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.0.98.62:35593)
0619 14:43:09.859613 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:09.859619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:09.859957 (+ 338us) server_negotiation.cc:407] Connection header received
0619 14:43:09.860169 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:09.860173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:09.860228 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:09.860312 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:09.860947 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.861789 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.864097 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.864443 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.864543 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:09.866552 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:09.866557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:09.866563 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:09.866592 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:09.868259 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.868730 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.868733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.868735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.868817 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.869160 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.869164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.869165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.869408 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:09.869483 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:09.869789 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 14:43:09.869896 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 14:43:09.870478 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.859319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.0.1:42480)
0619 14:43:09.859702 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:09.859888 (+ 186us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:09.860032 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:09.860452 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:09.860455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:09.860716 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:09.860829 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:09.860835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.863001 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.863004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:09.863973 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:09.863983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.864650 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.864653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:09.864728 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:09.865875 (+ 1147us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:09.865887 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:09.866405 (+ 518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:09.868395 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:09.868398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:09.868400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:09.868620 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:09.868930 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:09.868933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:09.868935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:09.869024 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:09.869553 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:09.869555 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:09.869635 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 14:43:09.870174 (+ 539us) client_negotiation.cc:240] Negotiation successful
0619 14:43:09.870303 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":4,"thread_start_us":117,"threads_started":1}
Jun 19 14:43:09 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880188, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 14:43:09.886070 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.874746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34175 (local address 127.0.0.1:48374)
0619 14:43:09.876003 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:09.876051 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:09.876220 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:09.876668 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:09.876674 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:09.877084 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:09.877200 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:09.877207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.878174 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.878177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:09.879125 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:09.879135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.879818 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.879821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:09.879927 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:09.880431 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:09.880444 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:09.882195 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:09.884210 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:09.884219 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:09.884226 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:09.884517 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:09.884808 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:09.884811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:09.884813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:09.884902 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:09.885419 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:09.885426 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:09.885555 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 14:43:09.885705 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 14:43:09.885898 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1092,"thread_start_us":89,"threads_started":1}
I20260619 14:43:09.886097 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.874779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48374 (local address 127.0.98.61:34175)
0619 14:43:09.874882 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:09.874886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:09.876121 (+ 1235us) server_negotiation.cc:407] Connection header received
0619 14:43:09.876374 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:09.876377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:09.876426 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:09.876488 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:09.877316 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.878068 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.879251 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.879613 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.879684 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:09.882328 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:09.882332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:09.882335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:09.882360 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:09.884086 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.884608 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.884613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.884615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.884704 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.884996 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.885002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.885007 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.885281 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:09.885417 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:09.885784 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 14:43:09.885915 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:43:09.888144 6518 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:34175: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a918948481984a82bef65cbedfad5ef7"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 34175
}
attrs {
promote: true
}
}
dest_uuid: "4f0cad194c5742c692d7152d92cd5565"
cas_config_opid_index: -1
I20260619 14:43:09.888852 6518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260619 14:43:09.890750 7079 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a918948481984a82bef65cbedfad5ef7 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: true } } }
I20260619 14:43:09.891492 7026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:09.893210 7079 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:34175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260619 14:43:09.893635 6984 master.cc:537] Master@127.0.98.61:34175 shutting down...
I20260619 14:43:09.893674 7026 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 1 FOLLOWER]: Refusing update from remote peer 4f0cad194c5742c692d7152d92cd5565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 14:43:09.894941 7079 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:09.894616 7080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4f0cad194c5742c692d7152d92cd5565" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: true } } }
I20260619 14:43:09.895678 7080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: This master's current role is: LEADER
W20260619 14:43:09.896895 6507 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 14:43:09.900012 6984 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:43:09.900321 6984 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:09.900441 6984 tablet_replica.cc:330] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: stopping tablet replica
I20260619 14:43:09.905462 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.905050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34175 (local address 127.0.0.1:48376)
0619 14:43:09.905268 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:09.905347 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260619 14:43:09.905648 6507 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 -> Peer a918948481984a82bef65cbedfad5ef7 (127.0.98.61:34175): Couldn't send request to peer a918948481984a82bef65cbedfad5ef7. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34175: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 14:43:09.906154 6984 master.cc:552] Master@127.0.98.61:34175 shutdown complete.
I20260619 14:43:09.906206 6984 master_runner.cc:305] Clearing existing system tablet
I20260619 14:43:09.906440 6984 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:09.906497 6984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:09.908448 6984 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:43:09.909409 7085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:09.909600 6984 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:09.909734 6984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:43:09.909777 6984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/wal
uuid: "a918948481984a82bef65cbedfad5ef7"
format_stamp: "Formatted at 2026-06-19 14:43:09 on dist-test-slave-cr22"
server_key: "f662bf7ef7f87592bdc2c41644c3eea0"
server_key_iv: "120c73685f0024b642bf989d79e9522a"
server_key_version: "encryptionkey@0"
I20260619 14:43:09.911660 6984 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260619 14:43:09.914314 6984 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 14:43:09.914419 6984 log.cc:1196] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 14:43:09.914465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44173
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35593,127.0.98.61:34175,127.0.98.60:44173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:43:09.914685 6984 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: Deleting consensus metadata
I20260619 14:43:09.914878 6984 master_runner.cc:315] Copying system tablet from 127.0.98.62:35593
I20260619 14:43:09.916213 6984 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35593
I20260619 14:43:09.924228 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.916834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42484 (local address 127.0.98.62:35593)
0619 14:43:09.916947 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:09.916952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:09.917159 (+ 207us) server_negotiation.cc:407] Connection header received
0619 14:43:09.917321 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:09.917324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:09.917378 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:09.917463 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:09.917875 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.918434 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.919269 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.919650 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.919783 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:09.920873 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:09.920876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:09.920879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:09.920905 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:09.922448 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.923004 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.923008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.923009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.923092 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:09.923349 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:09.923352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:09.923354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:09.923588 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:09.923686 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:09.923954 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 14:43:09.924062 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:43:09.924237 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:09.916739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.0.1:42484)
0619 14:43:09.917066 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:09.917078 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:09.917207 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:09.917479 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:09.917482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:09.917676 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:09.917755 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:09.917761 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.918544 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.918549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:09.919140 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:09.919148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:09.919786 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:09.919790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:09.919875 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:09.920408 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:09.920420 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:09.920728 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:09.922580 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:09.922583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:09.922585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:09.922848 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:09.923184 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:09.923186 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:09.923188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:09.923255 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:09.923686 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:09.923689 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:09.923787 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 14:43:09.923898 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 14:43:09.924045 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":91,"threads_started":1}
I20260619 14:43:09.924753 6540 tablet_copy_service.cc:140] P 4f0cad194c5742c692d7152d92cd5565: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a918948481984a82bef65cbedfad5ef7 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:42484)
I20260619 14:43:09.924937 6540 tablet_copy_service.cc:161] P 4f0cad194c5742c692d7152d92cd5565: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a918948481984a82bef65cbedfad5ef7 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:42484: session id = a918948481984a82bef65cbedfad5ef7-00000000000000000000000000000000
I20260619 14:43:09.926460 6540 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 14:43:09.928050 6984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 14:43:09.929853 6984 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: tablet copy: Starting download of 0 data blocks...
I20260619 14:43:09.930037 6984 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: tablet copy: Starting download of 1 WAL segments...
I20260619 14:43:09.931761 6984 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 14:43:09.933248 6540 tablet_copy_service.cc:342] P 4f0cad194c5742c692d7152d92cd5565: Request end of tablet copy session a918948481984a82bef65cbedfad5ef7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:42484
I20260619 14:43:09.933375 6540 tablet_copy_service.cc:433] P 4f0cad194c5742c692d7152d92cd5565: ending tablet copy session a918948481984a82bef65cbedfad5ef7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a918948481984a82bef65cbedfad5ef7
I20260619 14:43:09.940835 6984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:09.942804 7093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:09.942834 7096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:09.943048 7094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:09.943645 6984 server_base.cc:1034] running on GCE node
I20260619 14:43:09.943810 6984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:09.944190 6984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:09.945369 6984 hybrid_clock.cc:648] HybridClock initialized: now 1781880189945363 us; error 29 us; skew 500 ppm
Jun 19 14:43:09 dist-test-slave-cr22 krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880189, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:09.946858 6984 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 14:43:09.947316 6984 webserver.cc:460] Webserver started at http://127.0.98.61:35519/ using document root <none> and password file <none>
I20260619 14:43:09.947546 6984 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:09.947598 6984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:09.948856 6984 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 14:43:09.950111 7103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:09.950376 6984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 14:43:09.950464 6984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/wal
uuid: "a918948481984a82bef65cbedfad5ef7"
format_stamp: "Formatted at 2026-06-19 14:43:09 on dist-test-slave-cr22"
server_key: "f662bf7ef7f87592bdc2c41644c3eea0"
server_key_iv: "120c73685f0024b642bf989d79e9522a"
server_key_version: "encryptionkey@0"
I20260619 14:43:09.950564 6984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:09.958032 6984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:09.960134 6984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:09.965493 6984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34175
I20260619 14:43:09.966068 6984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-1/data/info.pb
I20260619 14:43:09.966459 7155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34175 every 8 connection(s)
I20260619 14:43:09.967700 7156 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:43:09.968554 7156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: Bootstrap starting.
I20260619 14:43:09.977144 7156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:43:09.977473 7156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: Bootstrap complete.
I20260619 14:43:09.977975 7156 raft_consensus.cc:348] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: true } }
I20260619 14:43:09.978112 7156 raft_consensus.cc:729] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a918948481984a82bef65cbedfad5ef7, State: Initialized, Role: LEARNER
I20260619 14:43:09.978250 7156 consensus_queue.cc:260] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: true } }
I20260619 14:43:09.978595 7159 sys_catalog.cc:455] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: true } } }
I20260619 14:43:09.978775 7159 sys_catalog.cc:458] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [sys.catalog]: This master's current role is: LEARNER
I20260619 14:43:09.978809 7156 sys_catalog.cc:564] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:43:09.981986 7170 catalog_manager.cc:1259] Loaded cluster ID: c48d93b0efcf4a02a9e36ec812ca32d4
I20260619 14:43:09.982097 7170 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: loading cluster ID for follower catalog manager: success
I20260619 14:43:09.983963 7170 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: acquiring CA information for follower catalog manager: success
I20260619 14:43:09.985158 7170 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 14:43:10.082002 7090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:10.082341 7090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:10.082424 7090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:10.086562 7090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:43:10.086648 7090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:10.086692 7090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:10.086730 7090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:43:10.086763 7090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:43:10.092258 7090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35593,127.0.98.61:34175,127.0.98.60:44173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:10.093362 7090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:10.094461 7090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:10.103340 7175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:10.103410 7178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:10.103374 7176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:10.103727 7090 server_base.cc:1034] running on GCE node
I20260619 14:43:10.104333 7090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:10.105515 7090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:10.106758 7090 hybrid_clock.cc:648] HybridClock initialized: now 1781880190106735 us; error 41 us; skew 500 ppm
Jun 19 14:43:10 dist-test-slave-cr22 krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880190, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:10.110188 7090 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 14:43:10.111223 7090 webserver.cc:460] Webserver started at http://127.0.98.60:41543/ using document root <none> and password file <none>
I20260619 14:43:10.111964 7090 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:10.112066 7090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:10.112314 7090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:10.114651 7090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/data/instance:
uuid: "83383d65b4da4ba5b62b800da9c65e9f"
format_stamp: "Formatted at 2026-06-19 14:43:10 on dist-test-slave-cr22"
server_key: "f30a60128b79d91e0d5f23183ce2fa9e"
server_key_iv: "47d04d39d52bcf9d2140bf0fae080923"
server_key_version: "encryptionkey@0"
I20260619 14:43:10.115307 7090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/wal/instance:
uuid: "83383d65b4da4ba5b62b800da9c65e9f"
format_stamp: "Formatted at 2026-06-19 14:43:10 on dist-test-slave-cr22"
server_key: "f30a60128b79d91e0d5f23183ce2fa9e"
server_key_iv: "47d04d39d52bcf9d2140bf0fae080923"
server_key_version: "encryptionkey@0"
I20260619 14:43:10.119361 7090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 14:43:10.122221 7185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:10.123711 7090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:43:10.123899 7090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/wal
uuid: "83383d65b4da4ba5b62b800da9c65e9f"
format_stamp: "Formatted at 2026-06-19 14:43:10 on dist-test-slave-cr22"
server_key: "f30a60128b79d91e0d5f23183ce2fa9e"
server_key_iv: "47d04d39d52bcf9d2140bf0fae080923"
server_key_version: "encryptionkey@0"
I20260619 14:43:10.124069 7090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:10.134616 7090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:10.137705 7090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:10.137902 7090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:10.145274 7090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44173
I20260619 14:43:10.145566 7237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44173 every 8 connection(s)
I20260619 14:43:10.146322 7090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/data/info.pb
I20260619 14:43:10.149502 7238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:10.153374 7238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:43:10.155615 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 7090
Jun 19 14:43:10 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880190, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 14:43:10 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 14:43:10.168190 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.155255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42496 (local address 127.0.98.62:35593)
0619 14:43:10.155404 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.155408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.156226 (+ 818us) server_negotiation.cc:407] Connection header received
0619 14:43:10.157468 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.157471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.157520 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.157774 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.158630 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.159246 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.160320 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.160589 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.160673 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.163894 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.163897 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.163899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.163924 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.165668 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.166331 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.166339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.166341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.166436 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.166879 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.166881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.166883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.167161 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.167248 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.167891 (+ 643us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.168021 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:43:10.168229 6711 heartbeater.cc:503] Master 127.0.98.62:35593 was elected leader, sending a full tablet report...
I20260619 14:43:10.169342 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.155568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.0.1:42496)
0619 14:43:10.156067 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.156109 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.157234 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.157803 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.157808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.158193 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:10.158445 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.158481 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.159384 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.159387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.160201 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.160209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.160738 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.160742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.160912 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.161748 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:10.161765 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:10.163664 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:10.165792 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.165800 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.165808 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.166205 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.166544 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.166547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.166549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.166718 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.167268 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:10.167277 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:10.167634 (+ 357us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.167805 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.168143 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":48,"spinlock_wait_cycles":1408,"thread_start_us":160,"threads_started":1}
I20260619 14:43:10.169343 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.156523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38846 (local address 127.0.98.60:44173)
0619 14:43:10.156865 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.156876 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.156913 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:43:10.157234 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.157256 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.157481 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.157624 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.158519 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.159471 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.160392 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.160851 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.160958 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.164274 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.164280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.164291 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.164312 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.166384 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.166955 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.166962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.166964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.167046 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.167322 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.167324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.167326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.167638 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.167835 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.168618 (+ 783us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.168759 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":254,"spinlock_wait_cycles":2944,"thread_start_us":146,"threads_started":1}
I20260619 14:43:10.170581 7238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:43:10 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880190, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 14:43:10.179814 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.171102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34175 (local address 127.0.0.1:48382)
0619 14:43:10.171243 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.171254 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.171316 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.171749 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.171752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.171912 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:10.171998 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.172002 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.172950 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.172956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.173493 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.173500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.174050 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.174053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.174154 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.174697 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:10.174709 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:10.176098 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:10.177830 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.177833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.177835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.178104 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.178415 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.178418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.178420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.178506 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.179185 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:10.179189 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:10.179293 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.179405 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.179620 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 14:43:10.179831 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.171214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.0.98.61:34175)
0619 14:43:10.171528 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.171533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.171546 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:43:10.171588 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.171591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.171639 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.171737 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.172102 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.172852 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.173602 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.173944 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.174045 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.176257 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.176261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.176263 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.176283 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.177701 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.178217 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.178220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.178221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.178295 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.178749 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.178753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.178755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.179025 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.179202 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.179505 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.179639 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":101,"threads_started":1}
I20260619 14:43:10.180711 7238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:43:10 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880190, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 14:43:10.188938 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.181390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38848 (local address 127.0.98.60:44173)
0619 14:43:10.181491 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.181494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.181506 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:43:10.181601 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.181603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.181646 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.181719 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.182033 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.182564 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.183325 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.183605 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.183704 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.185833 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.185837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.185839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.185858 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.187243 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.187748 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.187750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.187752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.187874 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.188146 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.188148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.188149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.188303 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.188378 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.188637 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.188736 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:43:10.188969 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.181288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44173 (local address 127.0.0.1:38848)
0619 14:43:10.181439 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.181452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.181515 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.181730 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.181732 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.181870 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:10.181940 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.181945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.182718 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.182720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.183223 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.183230 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.183730 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.183733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.183810 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.184237 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:10.184246 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:10.185587 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:10.187347 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.187349 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.187350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.187668 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.187980 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.187982 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.187984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.188063 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.188417 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:10.188420 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:10.188503 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.188592 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.188795 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":35}
I20260619 14:43:10.192477 7238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: Bootstrap starting.
I20260619 14:43:10.195107 7238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:10.195897 7238 log.cc:824] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:10.197899 7238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: No bootstrap required, opened a new log
I20260619 14:43:10.201113 7238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } }
I20260619 14:43:10.201370 7238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:10.201450 7238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83383d65b4da4ba5b62b800da9c65e9f, State: Initialized, Role: FOLLOWER
I20260619 14:43:10.201952 7238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } }
I20260619 14:43:10.202390 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } } }
I20260619 14:43:10.202522 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:43:10.203131 7238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:43:10.207768 7255 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:43:10.207854 7255 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:43:10.212641 7090 master_runner.cc:428] Detected that this master 83383d65b4da4ba5b62b800da9c65e9f is joining an existing cluster
I20260619 14:43:10.212697 7090 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:44173
I20260619 14:43:10.222761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 6582
I20260619 14:43:10.231954 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38627
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35593
--webserver_port=34923
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35593,127.0.98.61:34175
--tserver_master_addrs=127.0.98.62:35593,127.0.98.61:34175,127.0.98.60:44173 with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:43:10.247712 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.214608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44173 (local address 127.0.0.1:38860)
0619 14:43:10.214885 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.214901 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.214993 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.215386 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.215389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.215648 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:10.215822 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.215830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.216811 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.216814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.218301 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.218312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.218927 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.218933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.219031 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.219552 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:10.219566 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:10.220040 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:10.245866 (+ 25826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.245870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.245872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.246150 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.246455 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.246460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.246464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.246539 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.247047 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:10.247050 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:10.247148 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.247239 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.247396 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"spinlock_wait_cycles":768,"thread_start_us":122,"threads_started":1}
I20260619 14:43:10.249902 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.214893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.0.1:42512)
0619 14:43:10.215602 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.215615 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.215708 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.216057 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.216060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.216277 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:10.216406 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.216414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.217617 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.217621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.219895 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.219906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.220674 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.220677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.220785 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.221151 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:10.221166 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:10.244342 (+ 23176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:10.246665 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.246667 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.246669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.248193 (+ 1524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.248568 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.248574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.248580 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.248675 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.249248 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:10.249254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:10.249370 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.249463 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.249634 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":655,"thread_start_us":84,"threads_started":1}
I20260619 14:43:10.250514 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.214643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.0.98.62:35593)
0619 14:43:10.214790 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.214796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.215711 (+ 915us) server_negotiation.cc:407] Connection header received
0619 14:43:10.215781 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.215785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.215839 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.215930 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.216554 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.217463 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.220056 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.220430 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.220517 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.244614 (+ 24097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.244621 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.244624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.244672 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.246546 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.248336 (+ 1790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.248342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.248347 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.248449 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.248794 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.248809 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.248811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.249073 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.249173 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.250158 (+ 985us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.250300 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:43:10.253505 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.214964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38860 (local address 127.0.98.60:44173)
0619 14:43:10.215067 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.215072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.215082 (+ 10us) server_negotiation.cc:407] Connection header received
0619 14:43:10.215109 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.215112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.215150 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.215227 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.215935 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.216703 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.218400 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.218792 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.218908 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.220177 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.220182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.220187 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.220211 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.245712 (+ 25501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.246268 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.246272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.246275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.246356 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.246630 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.246633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.246636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.246915 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.247013 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.253167 (+ 6154us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.253297 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:43:10.258884 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.215302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.0.98.61:34175)
0619 14:43:10.215403 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.215408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.215973 (+ 565us) server_negotiation.cc:407] Connection header received
0619 14:43:10.216127 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.216130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.216176 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.216222 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.216801 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.217612 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.218998 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.219673 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.219755 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.247213 (+ 27458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.247219 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.247224 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.247262 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.249221 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.255136 (+ 5915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.255140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.255142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.255493 (+ 351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.257832 (+ 2339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.257836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.257841 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.258079 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.258194 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.258553 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.258692 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 14:43:10.259377 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.215310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34175 (local address 127.0.0.1:48394)
0619 14:43:10.215903 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.215917 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.216033 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.216346 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.216349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.216545 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:10.216658 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.216664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.217759 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.217761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.218873 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.218884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.219918 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.219921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.220020 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.220495 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:10.220508 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:10.247032 (+ 26524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:10.254491 (+ 7459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.254494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.254496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.254951 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.257536 (+ 2585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.257540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.257542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.257671 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.258264 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:10.258269 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:10.258374 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.259020 (+ 646us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.259191 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":68,"thread_start_us":79,"threads_started":1}
Jun 19 14:43:10 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880188, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 14:43:10.268003 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.259155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.0.98.60:44173)
0619 14:43:10.259311 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.259315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.259327 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:43:10.259401 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.259404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.259490 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.259572 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.260425 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.261208 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.262603 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.262841 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.262935 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.264987 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.264990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.264992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.265012 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.266431 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.266857 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.266865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.266867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.266932 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.267193 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.267198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.267200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.267362 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.267481 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.267750 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.267850 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 14:43:10.268003 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.258938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44173 (local address 127.0.0.1:38872)
0619 14:43:10.259088 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.259104 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.259199 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.259707 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.259713 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.259906 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:10.260024 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.260036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.261352 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.261358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.262477 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.262487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.263029 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.263032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.263098 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.263602 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:10.263614 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:10.264848 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:10.266551 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.266554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.266556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.266757 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.267017 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.267021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.267023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.267106 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.267486 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:10.267490 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:10.267585 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.267694 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.267844 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 14:43:10.269192 6518 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:44173: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 44173
}
attrs {
promote: true
}
}
dest_uuid: "4f0cad194c5742c692d7152d92cd5565"
cas_config_opid_index: 5
I20260619 14:43:10.269717 6518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: true } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: true } }
I20260619 14:43:10.270756 7079 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 83383d65b4da4ba5b62b800da9c65e9f (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: true } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: true } } }
I20260619 14:43:10.272318 7079 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:44173
I20260619 14:43:10.272497 7079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4f0cad194c5742c692d7152d92cd5565" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: true } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: true } } }
I20260619 14:43:10.272593 7079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:10.273332 7212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:10.274693 7212 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [term 1 FOLLOWER]: Refusing update from remote peer 4f0cad194c5742c692d7152d92cd5565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 14:43:10.275344 7090 master.cc:537] Master@127.0.98.60:44173 shutting down...
I20260619 14:43:10.276603 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.270875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34175 (local address 127.0.0.1:48410)
0619 14:43:10.271702 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.271715 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.271784 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.272220 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.272223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.272242 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:10.272329 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.272338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.273525 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.273528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.274715 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.274727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.276131 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.276135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.276232 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.276273 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.276348 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.276430 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767}
I20260619 14:43:10.277182 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.271110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.0.98.61:34175)
0619 14:43:10.271315 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.271320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.271865 (+ 545us) server_negotiation.cc:407] Connection header received
0619 14:43:10.271919 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.271923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.271980 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.272067 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:10.272527 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.273282 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.274920 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.275590 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.275675 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.275849 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.276927 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.277021 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260619 14:43:10.276801 7080 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:10.278048 7121 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 1 LEARNER]: Refusing update from remote peer 4f0cad194c5742c692d7152d92cd5565: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 14:43:10.278632 7080 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260619 14:43:10.279939 6506 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 -> Peer 83383d65b4da4ba5b62b800da9c65e9f (127.0.98.60:44173): Couldn't send request to peer 83383d65b4da4ba5b62b800da9c65e9f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 14:43:10.280633 7121 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 83383d65b4da4ba5b62b800da9c65e9f (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: true } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: true } } }
I20260619 14:43:10.282356 7090 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:43:10.282619 7090 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:10.282752 7090 tablet_replica.cc:330] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: stopping tablet replica
I20260619 14:43:10.283603 7159 sys_catalog.cc:455] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f0cad194c5742c692d7152d92cd5565. Latest consensus state: current_term: 1 leader_uuid: "4f0cad194c5742c692d7152d92cd5565" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: true } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: true } } }
I20260619 14:43:10.283720 7159 sys_catalog.cc:458] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [sys.catalog]: This master's current role is: LEARNER
I20260619 14:43:10.283808 7159 sys_catalog.cc:455] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f0cad194c5742c692d7152d92cd5565" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: true } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: true } } }
I20260619 14:43:10.283866 7159 sys_catalog.cc:458] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [sys.catalog]: This master's current role is: LEARNER
I20260619 14:43:10.284793 7079 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565: attempting to promote NON_VOTER a918948481984a82bef65cbedfad5ef7 to VOTER
I20260619 14:43:10.285229 7079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: true } }
I20260619 14:43:10.286595 7121 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 1 LEARNER]: Refusing update from remote peer 4f0cad194c5742c692d7152d92cd5565: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 14:43:10.287210 7080 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260619 14:43:10.287328 6506 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 -> Peer 83383d65b4da4ba5b62b800da9c65e9f (127.0.98.60:44173): Couldn't send request to peer 83383d65b4da4ba5b62b800da9c65e9f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 14:43:10.289242 7079 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a918948481984a82bef65cbedfad5ef7 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: true } } }
I20260619 14:43:10.289769 7121 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a918948481984a82bef65cbedfad5ef7 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: true } } }
I20260619 14:43:10.290796 7079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4f0cad194c5742c692d7152d92cd5565" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: true } } }
I20260619 14:43:10.290931 7079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:10.291267 7080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f0cad194c5742c692d7152d92cd5565" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: true } } }
I20260619 14:43:10.291378 7080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:10.292255 7266 sys_catalog.cc:455] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f0cad194c5742c692d7152d92cd5565" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: true } } }
I20260619 14:43:10.292379 7266 sys_catalog.cc:458] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:43:10.303655 7090 master.cc:552] Master@127.0.98.60:44173 shutdown complete.
I20260619 14:43:10.303737 7090 master_runner.cc:305] Clearing existing system tablet
I20260619 14:43:10.304049 7090 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:10.304122 7090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:10.305653 7090 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:43:10.306577 7269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:10.307718 7090 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:10.308130 7090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:43:10.308203 7090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/wal
uuid: "83383d65b4da4ba5b62b800da9c65e9f"
format_stamp: "Formatted at 2026-06-19 14:43:10 on dist-test-slave-cr22"
server_key: "f30a60128b79d91e0d5f23183ce2fa9e"
server_key_iv: "47d04d39d52bcf9d2140bf0fae080923"
server_key_version: "encryptionkey@0"
I20260619 14:43:10.310367 7090 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:10.312696 7090 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 14:43:10.312804 7090 log.cc:1196] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 14:43:10.313066 7090 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: Deleting consensus metadata
I20260619 14:43:10.313253 7090 master_runner.cc:315] Copying system tablet from 127.0.98.62:35593
I20260619 14:43:10.314538 7090 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35593
I20260619 14:43:10.323185 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.315151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.0.1:42524)
0619 14:43:10.315540 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.315555 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.315696 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.316003 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.316007 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.316217 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:10.316330 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.316335 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.317185 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.317192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.317866 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.317875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.318408 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.318412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.318522 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.319180 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:10.319193 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:10.319641 (+ 448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:10.321528 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.321532 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.321534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.321758 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.322069 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.322072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.322074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.322164 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.322566 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:10.322569 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:10.322684 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.322797 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.322970 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":88,"threads_started":1}
I20260619 14:43:10.323173 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.315225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42524 (local address 127.0.98.62:35593)
0619 14:43:10.315368 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.315374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.315647 (+ 273us) server_negotiation.cc:407] Connection header received
0619 14:43:10.315817 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.315820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.315879 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.315979 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.316457 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.317058 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.318017 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.318282 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.318405 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.319802 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.319805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.319808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.319836 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.321389 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.321869 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.321874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.321876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.321964 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.322277 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.322280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.322282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.322468 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.322578 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.322867 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.322988 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:43:10.324851 6540 tablet_copy_service.cc:140] P 4f0cad194c5742c692d7152d92cd5565: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 83383d65b4da4ba5b62b800da9c65e9f ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:42524)
I20260619 14:43:10.324963 6540 tablet_copy_service.cc:161] P 4f0cad194c5742c692d7152d92cd5565: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 83383d65b4da4ba5b62b800da9c65e9f at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:42524: session id = 83383d65b4da4ba5b62b800da9c65e9f-00000000000000000000000000000000
I20260619 14:43:10.325868 6540 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 14:43:10.327157 7090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 14:43:10.329388 7090 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: tablet copy: Starting download of 0 data blocks...
I20260619 14:43:10.329595 7090 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: tablet copy: Starting download of 1 WAL segments...
I20260619 14:43:10.331096 7090 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 14:43:10.332980 6540 tablet_copy_service.cc:342] P 4f0cad194c5742c692d7152d92cd5565: Request end of tablet copy session 83383d65b4da4ba5b62b800da9c65e9f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:42524
I20260619 14:43:10.333104 6540 tablet_copy_service.cc:433] P 4f0cad194c5742c692d7152d92cd5565: ending tablet copy session 83383d65b4da4ba5b62b800da9c65e9f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 83383d65b4da4ba5b62b800da9c65e9f
I20260619 14:43:10.340016 7090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:10.341941 7276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:10.342092 7279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:10.343588 7090 server_base.cc:1034] running on GCE node
W20260619 14:43:10.347625 7277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:10.347900 7090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:10.348242 7090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:10.349387 7090 hybrid_clock.cc:648] HybridClock initialized: now 1781880190349377 us; error 32 us; skew 500 ppm
Jun 19 14:43:10 dist-test-slave-cr22 krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880190, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:10.351095 7090 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 14:43:10.351718 7090 webserver.cc:460] Webserver started at http://127.0.98.60:35209/ using document root <none> and password file <none>
I20260619 14:43:10.351917 7090 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:10.351964 7090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:10.353066 7090 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:43:10.353982 7286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:10.354214 7090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:43:10.354305 7090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/wal
uuid: "83383d65b4da4ba5b62b800da9c65e9f"
format_stamp: "Formatted at 2026-06-19 14:43:10 on dist-test-slave-cr22"
server_key: "f30a60128b79d91e0d5f23183ce2fa9e"
server_key_iv: "47d04d39d52bcf9d2140bf0fae080923"
server_key_version: "encryptionkey@0"
I20260619 14:43:10.354403 7090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:10.358697 7090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:10.360708 7090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:10.365746 7090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44173
I20260619 14:43:10.366257 7090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/master-2/data/info.pb
I20260619 14:43:10.366680 7338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44173 every 8 connection(s)
I20260619 14:43:10.367867 7339 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:43:10.369010 7339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: Bootstrap starting.
I20260619 14:43:10.380205 7339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:43:10.380503 7339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: Bootstrap complete.
I20260619 14:43:10.381232 7339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: true } }
I20260619 14:43:10.381377 7339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83383d65b4da4ba5b62b800da9c65e9f, State: Initialized, Role: LEARNER
I20260619 14:43:10.381493 7339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: true } }
I20260619 14:43:10.381860 7342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: true } } }
I20260619 14:43:10.381999 7339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:43:10.382074 7342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [sys.catalog]: This master's current role is: LEARNER
I20260619 14:43:10.384730 6845 heartbeater.cc:503] Master 127.0.98.62:35593 was elected leader, sending a full tablet report...
I20260619 14:43:10.385056 7353 catalog_manager.cc:1259] Loaded cluster ID: c48d93b0efcf4a02a9e36ec812ca32d4
I20260619 14:43:10.385139 7353 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: loading cluster ID for follower catalog manager: success
I20260619 14:43:10.387284 7353 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: acquiring CA information for follower catalog manager: success
I20260619 14:43:10.388651 7353 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 14:43:10.390646 7264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:10.390861 7264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:10.390933 7264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:10.395205 7264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:10.395260 7264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:10.395362 7264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:43:10.400633 7264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34923
--tserver_master_addrs=127.0.98.62:35593,127.0.98.61:34175,127.0.98.60:44173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:10.401620 7264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:10.402678 7264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:10.411986 7361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:10.411942 7358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:10.412222 7359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:10.412860 7264 server_base.cc:1034] running on GCE node
I20260619 14:43:10.413260 7264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:10.413911 7264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:10.415081 7264 hybrid_clock.cc:648] HybridClock initialized: now 1781880190415050 us; error 42 us; skew 500 ppm
Jun 19 14:43:10 dist-test-slave-cr22 krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880190, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:10.418183 7264 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 14:43:10.419023 7264 webserver.cc:460] Webserver started at http://127.0.98.1:34923/ using document root <none> and password file <none>
I20260619 14:43:10.419612 7264 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:10.419687 7264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:10.423374 7264 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:43:10.425433 7368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:10.426666 7264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:43:10.426795 7264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "bf2f8ba8faff4d7dab0982c9b75bf143"
format_stamp: "Formatted at 2026-06-19 14:43:09 on dist-test-slave-cr22"
server_key: "f9e6a25529d6bee14757c0a978cb56ac"
server_key_iv: "c77b302c87a670c8c5ea106ef825ad70"
server_key_version: "encryptionkey@0"
I20260619 14:43:10.427472 7264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:10.441461 7264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:10.444502 7264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:10.444695 7264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:10.445259 7264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:10.446079 7264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:10.446146 7264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:10.446213 7264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:10.446254 7264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:10.455770 7264 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38627
I20260619 14:43:10.455814 7481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38627 every 8 connection(s)
I20260619 14:43:10.456869 7264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:43:10.461354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 7264
I20260619 14:43:10.461591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 6716
Jun 19 14:43:10 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880190, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:43:10.469483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34115
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35593
--webserver_port=44361
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35593,127.0.98.61:34175
--tserver_master_addrs=127.0.98.62:35593,127.0.98.61:34175,127.0.98.60:44173 with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:43:10 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880190, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 14:43:10 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880190, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 14:43:10.475253 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.458630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33763 (local address 127.0.98.60:44173)
0619 14:43:10.459014 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.459018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.460292 (+ 1274us) server_negotiation.cc:407] Connection header received
0619 14:43:10.461135 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.461146 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.461203 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.461273 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.464032 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.465308 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.466782 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.467251 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.467355 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.471040 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.471044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.471047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.471082 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.472995 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.473546 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.473550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.473552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.473636 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.473972 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.473976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.473980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.474211 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.474296 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.474904 (+ 608us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.475043 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":79,"threads_started":1}
I20260619 14:43:10.475912 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.458708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36299 (local address 127.0.98.62:35593)
0619 14:43:10.458862 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.458867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.459827 (+ 960us) server_negotiation.cc:407] Connection header received
0619 14:43:10.461136 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.461141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.461203 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.461289 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.463358 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.464198 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.466017 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.466349 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.466427 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.471949 (+ 5522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.471953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.471957 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.471995 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.473955 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.474421 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.474425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.474427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.474515 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.474842 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.474845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.474847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.475052 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.475121 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.475665 (+ 544us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.475780 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:43:10.476089 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.460098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42857 (local address 127.0.98.61:34175)
0619 14:43:10.460240 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.460247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.460344 (+ 97us) server_negotiation.cc:407] Connection header received
0619 14:43:10.461078 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.461082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.461146 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.461218 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.464057 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.464803 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.466016 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.466364 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.466431 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.470584 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.470588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.470591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.470626 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.472566 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.473212 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.473218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.473223 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.473314 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.473724 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.473730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.473734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.473996 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.474099 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.475741 (+ 1642us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.475886 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:43:10.476430 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.459049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44173 (local address 127.0.98.1:33763)
0619 14:43:10.460186 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.460200 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.460921 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.461758 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.461767 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.462245 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:10.463939 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.463945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.465452 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.465458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.466631 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.466644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.467564 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.467570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.467730 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.468213 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:10.468238 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:10.470890 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:10.473124 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.473131 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.473139 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.473419 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.473755 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.473761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.473764 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.473866 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.474352 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:10.474358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:10.474570 (+ 212us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.475418 (+ 848us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.475618 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"mutex_wait_us":1151,"spinlock_wait_cycles":26624,"thread_start_us":800,"threads_started":1}
I20260619 14:43:10.476495 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.459048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.1:36299)
0619 14:43:10.459608 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.459675 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.460921 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.462264 (+ 1343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.462267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.462429 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:10.463175 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.463197 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.464353 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.464356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.465878 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.465888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.466630 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.466634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.466989 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.467849 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:10.467874 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:10.471299 (+ 3425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:10.474069 (+ 2770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.474071 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.474073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.474312 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.474650 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.474654 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.474657 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.474739 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.475192 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:10.475194 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:10.475278 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.476144 (+ 866us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.476292 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":241,"spinlock_wait_cycles":31232,"thread_start_us":107,"threads_started":1}
I20260619 14:43:10.477766 7483 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35593
I20260619 14:43:10.477913 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.459985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34175 (local address 127.0.98.1:42857)
0619 14:43:10.460230 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.460249 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.460886 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.462576 (+ 1690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.462579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.462722 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:10.463791 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.463798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.464945 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.464948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.465899 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.465905 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.466659 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.466662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.466992 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.468210 (+ 1218us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:10.468224 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:10.470406 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:10.472681 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.472690 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.472711 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.473078 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.473418 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.473424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.473429 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.473616 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.474157 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:10.474168 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:10.474572 (+ 404us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.474829 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.475253 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":1120,"spinlock_wait_cycles":45568,"thread_start_us":94,"threads_started":1}
I20260619 14:43:10.478034 7483 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:10.478065 7482 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44173
I20260619 14:43:10.478155 7482 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:10.478652 7482 heartbeater.cc:511] Master 127.0.98.60:44173 requested a full tablet report, sending...
I20260619 14:43:10.479012 7484 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34175
I20260619 14:43:10.479097 7484 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:10.479220 7484 heartbeater.cc:511] Master 127.0.98.61:34175 requested a full tablet report, sending...
I20260619 14:43:10.478664 7483 heartbeater.cc:511] Master 127.0.98.62:35593 requested a full tablet report, sending...
I20260619 14:43:10.480360 7117 ts_manager.cc:194] Registered new tserver with Master: bf2f8ba8faff4d7dab0982c9b75bf143 (127.0.98.1:38627)
I20260619 14:43:10.480465 7299 ts_manager.cc:194] Registered new tserver with Master: bf2f8ba8faff4d7dab0982c9b75bf143 (127.0.98.1:38627)
I20260619 14:43:10.481143 6518 ts_manager.cc:194] Re-registered known tserver with Master: bf2f8ba8faff4d7dab0982c9b75bf143 (127.0.98.1:38627)
I20260619 14:43:10.481760 6518 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36299
W20260619 14:43:10.591495 7490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:10.591777 7490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:10.591847 7490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:10.595851 7490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:10.595937 7490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:10.596022 7490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:43:10.601186 7490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44361
--tserver_master_addrs=127.0.98.62:35593,127.0.98.61:34175,127.0.98.60:44173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:10.602254 7490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:10.602525 6979 heartbeater.cc:503] Master 127.0.98.62:35593 was elected leader, sending a full tablet report...
I20260619 14:43:10.603513 7490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:10.616437 7496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:10.616540 7495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:10.616782 7490 server_base.cc:1034] running on GCE node
W20260619 14:43:10.616997 7498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:10.617499 7490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:10.618255 7490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:10.619423 7490 hybrid_clock.cc:648] HybridClock initialized: now 1781880190619396 us; error 53 us; skew 500 ppm
Jun 19 14:43:10 dist-test-slave-cr22 krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880190, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:10.623646 7490 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 14:43:10.625052 7490 webserver.cc:460] Webserver started at http://127.0.98.2:44361/ using document root <none> and password file <none>
I20260619 14:43:10.626070 7490 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:10.626178 7490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:10.631309 7490 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:43:10.633872 7505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:10.635437 7490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:43:10.635586 7490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "23ad98e2a58d4320aaf58238a845e40a"
format_stamp: "Formatted at 2026-06-19 14:43:09 on dist-test-slave-cr22"
server_key: "a98d7dd7f4f2ba3b34cfc71738cde456"
server_key_iv: "cd679f0cacffd57d45fd376d9093e030"
server_key_version: "encryptionkey@0"
I20260619 14:43:10.636251 7490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:10.653671 7490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:10.657358 7490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:10.657554 7490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:10.658146 7490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:10.659430 7490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:10.659539 7490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:10.659608 7490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:10.659638 7490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:10.669005 7490 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34115
I20260619 14:43:10.669028 7618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34115 every 8 connection(s)
I20260619 14:43:10.670004 7490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:43:10.677559 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.672105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44173 (local address 127.0.0.1:38880)
0619 14:43:10.672262 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.672277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.672379 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.673773 (+ 1394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.673777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.673800 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:10.673916 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.673924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.674997 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.675002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.676203 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.676213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.677148 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.677151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.677243 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.677270 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.677325 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.677403 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 14:43:10.677556 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 7490
I20260619 14:43:10.677721 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 6850
I20260619 14:43:10.678298 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.672722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38880 (local address 127.0.98.60:44173)
0619 14:43:10.673481 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.673487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.673501 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:43:10.673561 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.673564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.673622 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.673740 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:10.674057 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.674865 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.676336 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.677000 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.677767 (+ 767us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.677960 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.678002 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.678095 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":93,"threads_started":1}
I20260619 14:43:10.678972 7309 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 14:43:10.679545 7342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f0cad194c5742c692d7152d92cd5565. Latest consensus state: current_term: 1 leader_uuid: "4f0cad194c5742c692d7152d92cd5565" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: true } } }
I20260619 14:43:10.679699 7342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [sys.catalog]: This master's current role is: LEARNER
Jun 19 14:43:10 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880190, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 14:43:10 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880190, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 14:43:10.686379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33247
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35593
--webserver_port=40861
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35593,127.0.98.61:34175
--tserver_master_addrs=127.0.98.62:35593,127.0.98.61:34175,127.0.98.60:44173 with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:43:10.688256 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.673714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50133 (local address 127.0.98.61:34175)
0619 14:43:10.673848 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.673853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.674070 (+ 217us) server_negotiation.cc:407] Connection header received
0619 14:43:10.674268 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.674271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.674318 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.674377 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.676236 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.677132 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.680040 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.680459 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.680580 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.684282 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.684287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.684291 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.684323 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.686261 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.686928 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.686931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.686933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.687008 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.687345 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.687349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.687350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.687580 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.687665 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.687975 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.688102 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:43:10.688370 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.671743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45335 (local address 127.0.98.62:35593)
0619 14:43:10.671867 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.671872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.673465 (+ 1593us) server_negotiation.cc:407] Connection header received
0619 14:43:10.674143 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.674147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.674203 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.674261 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.676235 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.677132 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.680080 (+ 2948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.680314 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.680435 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.683840 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.683843 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.683846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.683874 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.685702 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.686374 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.686378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.686380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.686462 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.686897 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.686904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.686908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.687145 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.687226 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.688017 (+ 791us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.688156 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 14:43:10 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880190, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 14:43:10.689381 7626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.673366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34175 (local address 127.0.98.2:50133)
0619 14:43:10.673986 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.673999 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.674151 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.674585 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.674594 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.675262 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:10.676074 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.676080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.677287 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.677291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.679876 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.679886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.680926 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.680929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.681000 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.681471 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:10.681484 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:10.684126 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:10.686372 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.686375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.686377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.686821 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.687128 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.687132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.687134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.687220 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.687719 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:10.687722 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:10.687803 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.688768 (+ 965us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.688905 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":98,"spinlock_wait_cycles":57856,"thread_start_us":78,"threads_started":1}
I20260619 14:43:10.689677 7625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.672329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.2:45335)
0619 14:43:10.673368 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.673385 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.673917 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.674585 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.674594 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.675260 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:10.676070 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.676080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.677287 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.677291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.679428 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.679478 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.680470 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.680474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.680660 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.681274 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:10.681316 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:10.683674 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:10.685861 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.685868 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.685887 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.686252 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.686585 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.686587 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.686589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.686778 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.687326 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:10.687334 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:10.687700 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.687936 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.688266 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":878,"spinlock_wait_cycles":48512,"thread_start_us":162,"threads_started":1}
I20260619 14:43:10.690881 7620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35593
I20260619 14:43:10.691082 7620 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:10.691721 7620 heartbeater.cc:511] Master 127.0.98.62:35593 requested a full tablet report, sending...
I20260619 14:43:10.692986 6518 ts_manager.cc:194] Re-registered known tserver with Master: 23ad98e2a58d4320aaf58238a845e40a (127.0.98.2:34115)
I20260619 14:43:10.693508 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.672328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44173 (local address 127.0.98.2:33417)
0619 14:43:10.672861 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.672911 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.673912 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.674584 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.674592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.675257 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:10.676059 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.676076 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.677172 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.677175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.686884 (+ 9709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.686898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.687819 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.687824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.687937 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.688438 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:10.688453 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:10.689749 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:10.691925 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.691928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.691929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.692137 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.692407 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.692411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.692413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.692506 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.692953 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:10.692956 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:10.693076 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.693184 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.693321 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":104,"spinlock_wait_cycles":7424,"thread_start_us":799,"threads_started":1}
I20260619 14:43:10.693519 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.671958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33417 (local address 127.0.98.60:44173)
0619 14:43:10.672111 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.672118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.673016 (+ 898us) server_negotiation.cc:407] Connection header received
0619 14:43:10.674118 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.674121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.674166 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.674214 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.676214 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.677000 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.687038 (+ 10038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.687502 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.687619 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.689930 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.689934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.689937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.689966 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.691774 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.692223 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.692228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.692230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.692312 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.692604 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.692607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.692609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.692843 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.692953 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.693244 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.693382 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:43:10.693845 6518 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45335
I20260619 14:43:10.694202 7619 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44173
I20260619 14:43:10.694299 7621 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34175
I20260619 14:43:10.694345 7619 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:10.694613 7619 heartbeater.cc:511] Master 127.0.98.60:44173 requested a full tablet report, sending...
I20260619 14:43:10.694386 7621 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:10.695005 7621 heartbeater.cc:511] Master 127.0.98.61:34175 requested a full tablet report, sending...
I20260619 14:43:10.695569 7299 ts_manager.cc:194] Registered new tserver with Master: 23ad98e2a58d4320aaf58238a845e40a (127.0.98.2:34115)
I20260619 14:43:10.696396 7117 ts_manager.cc:194] Registered new tserver with Master: 23ad98e2a58d4320aaf58238a845e40a (127.0.98.2:34115)
W20260619 14:43:10.807287 7627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:10.807626 7627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:10.807682 7627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:10.811810 7627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:10.811867 7627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:10.811926 7627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:43:10.817286 7627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40861
--tserver_master_addrs=127.0.98.62:35593,127.0.98.61:34175,127.0.98.60:44173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:10.818642 7627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:10.819859 7627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:10.829536 7632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:10.829741 7635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:10.829787 7627 server_base.cc:1034] running on GCE node
W20260619 14:43:10.829679 7633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:10.830334 7627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:10.831063 7627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:10.832266 7627 hybrid_clock.cc:648] HybridClock initialized: now 1781880190832240 us; error 43 us; skew 500 ppm
Jun 19 14:43:10 dist-test-slave-cr22 krb5kdc[6475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880190, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:10.835654 7627 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 14:43:10.836658 7627 webserver.cc:460] Webserver started at http://127.0.98.3:40861/ using document root <none> and password file <none>
I20260619 14:43:10.837308 7627 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:10.837432 7627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:10.841706 7627 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:43:10.843767 7642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:10.845002 7627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:43:10.845127 7627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "875fc9e2d4544467a24fce99d075ed32"
format_stamp: "Formatted at 2026-06-19 14:43:09 on dist-test-slave-cr22"
server_key: "585ee5c8df625d74e23971051adc5fee"
server_key_iv: "1b4bcb288be9a342f520d5c9f3987bb0"
server_key_version: "encryptionkey@0"
I20260619 14:43:10.845669 7627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:10.880178 7627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:10.883948 7627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:10.884181 7627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:10.884963 7627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:10.885921 7627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:10.885993 7627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:10.886065 7627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:10.886096 7627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:10.895838 7627 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33247
I20260619 14:43:10.895855 7755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33247 every 8 connection(s)
I20260619 14:43:10.896968 7627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:43:10.906463 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 7627
Jun 19 14:43:10 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880190, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 14:43:10 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880190, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 14:43:10 dist-test-slave-cr22 krb5kdc[6475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880190, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 14:43:10.914031 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.898927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34903 (local address 127.0.98.60:44173)
0619 14:43:10.899043 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.899047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.900057 (+ 1010us) server_negotiation.cc:407] Connection header received
0619 14:43:10.901233 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.901237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.901323 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.901390 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.903005 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.903618 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.905245 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.905576 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.905682 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.909857 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.909861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.909865 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.909893 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.911622 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.912228 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.912235 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.912240 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.912315 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.912718 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.912721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.912723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.912945 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.913014 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.913754 (+ 740us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.913866 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:43:10.915503 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.899216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44173 (local address 127.0.98.3:34903)
0619 14:43:10.899842 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.899910 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.901003 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.901545 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.901551 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.902213 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:10.902837 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.902844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.903744 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.903747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.905126 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.905136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.905683 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.905687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.905795 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.907542 (+ 1747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:10.907557 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:10.909716 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:10.911748 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.911754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.911771 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.912107 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.912432 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.912436 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.912438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.912607 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.913092 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:10.913098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:10.913441 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.914176 (+ 735us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.914334 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":283,"spinlock_wait_cycles":52352,"thread_start_us":726,"threads_started":1}
I20260619 14:43:10.915701 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.900309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55647 (local address 127.0.98.61:34175)
0619 14:43:10.900435 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.900439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.900820 (+ 381us) server_negotiation.cc:407] Connection header received
0619 14:43:10.901708 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.901711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.901749 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.901903 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.902964 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.903827 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.905268 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.905546 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.905624 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.909712 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.909716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.909720 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.909746 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.912068 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.912534 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.912537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.912539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.912614 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.912899 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.912903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.912905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.913136 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.913219 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.915418 (+ 2199us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.915554 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:43:10.916819 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.900205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34175 (local address 127.0.98.3:55647)
0619 14:43:10.900699 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.900713 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.901029 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.901913 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.901916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.902231 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:10.902824 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.902838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.903951 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.903954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.905137 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.905143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.905794 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.905797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.905896 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.906755 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:10.906775 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:10.909542 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:10.912188 (+ 2646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.912193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.912198 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.912432 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.912721 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.912724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.912725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.912804 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.913258 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:10.913260 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:10.913441 (+ 181us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.913697 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.914029 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":95,"spinlock_wait_cycles":4224,"thread_start_us":95,"threads_started":1}
I20260619 14:43:10.917254 7756 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44173
I20260619 14:43:10.917519 7756 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:10.918105 7756 heartbeater.cc:511] Master 127.0.98.60:44173 requested a full tablet report, sending...
I20260619 14:43:10.919071 7758 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34175
I20260619 14:43:10.919147 7758 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:10.919301 7758 heartbeater.cc:511] Master 127.0.98.61:34175 requested a full tablet report, sending...
I20260619 14:43:10.919793 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.908682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38884 (local address 127.0.98.60:44173)
0619 14:43:10.908805 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.908826 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.909647 (+ 821us) server_negotiation.cc:407] Connection header received
0619 14:43:10.909814 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.909817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.909864 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.909924 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.910925 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.911722 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.913068 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.913406 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.913489 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.915623 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.915626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.915628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.915650 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.917852 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.918367 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.918373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.918374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.918457 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.918785 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.918788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.918791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.919002 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.919124 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.919522 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.919643 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:43:10.920426 7117 ts_manager.cc:194] Registered new tserver with Master: 875fc9e2d4544467a24fce99d075ed32 (127.0.98.3:33247)
I20260619 14:43:10.920593 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.908466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42530 (local address 127.0.98.62:35593)
0619 14:43:10.909232 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.909239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.909495 (+ 256us) server_negotiation.cc:407] Connection header received
0619 14:43:10.909615 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.909621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.909681 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.909766 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.910876 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.911715 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.913069 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.913423 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.913507 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.915600 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.915604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.915607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.915634 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.918117 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.918619 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.918624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.918625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.918719 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.919025 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.919029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.919034 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.919269 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.919394 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.920291 (+ 897us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.920408 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":81,"threads_started":1}
I20260619 14:43:10.921532 7299 ts_manager.cc:194] Registered new tserver with Master: 875fc9e2d4544467a24fce99d075ed32 (127.0.98.3:33247)
I20260619 14:43:10.922492 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.898819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41809 (local address 127.0.98.62:35593)
0619 14:43:10.898958 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.898962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.900557 (+ 1595us) server_negotiation.cc:407] Connection header received
0619 14:43:10.901511 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.901514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.901555 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.901616 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.903055 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.903865 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.905083 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.905374 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.905472 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.916178 (+ 10706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.916184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.916188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.916216 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.918649 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.919130 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.919136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.919141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.919226 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.919564 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.919567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.919569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.919756 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.919841 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.922214 (+ 2373us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.922337 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:43:10.923022 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.899216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.3:41809)
0619 14:43:10.900458 (+ 1242us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.900472 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.901003 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.901656 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.901661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.902224 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:10.902836 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.902844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.904011 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.904014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.904958 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.904967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.905515 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.905519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.905714 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.906411 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:10.906427 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:10.910828 (+ 4401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:10.918770 (+ 7942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.918774 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.918776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.919023 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.919338 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:10.919341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:10.919343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:10.919462 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:10.919974 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:10.919977 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:10.920082 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.922674 (+ 2592us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.922834 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160,"mutex_wait_us":305,"spinlock_wait_cycles":42368,"thread_start_us":182,"threads_started":1}
I20260619 14:43:10.923591 7757 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35593
I20260619 14:43:10.923678 7757 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:10.923856 7757 heartbeater.cc:511] Master 127.0.98.62:35593 requested a full tablet report, sending...
I20260619 14:43:10.924481 6520 ts_manager.cc:194] Re-registered known tserver with Master: 875fc9e2d4544467a24fce99d075ed32 (127.0.98.3:33247)
I20260619 14:43:10.925223 6520 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41809
I20260619 14:43:10.923629 6518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:43:10.927839 6518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:43:10.930371 7267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f0cad194c5742c692d7152d92cd5565" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: true } } }
I20260619 14:43:10.930517 7267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:10.931183 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.908943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48414 (local address 127.0.98.61:34175)
0619 14:43:10.909195 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.909199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.920197 (+ 10998us) server_negotiation.cc:407] Connection header received
0619 14:43:10.920392 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.920396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.920447 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.920531 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:10.923687 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.924472 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.925580 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.925873 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.925980 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.927561 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:10.927564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:10.927566 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:10.927591 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:10.929123 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.929596 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.929600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.929602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.929696 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:10.930090 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:10.930093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:10.930096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:10.930340 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:10.930448 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.930845 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.930982 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
I20260619 14:43:10.934398 7774 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565: attempting to promote NON_VOTER 83383d65b4da4ba5b62b800da9c65e9f to VOTER
I20260619 14:43:10.934821 7774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: false } }
I20260619 14:43:10.936453 7309 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [term 1 LEARNER]: Refusing update from remote peer 4f0cad194c5742c692d7152d92cd5565: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 14:43:10.936796 7773 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 14:43:10.938271 7121 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 1 FOLLOWER]: Refusing update from remote peer 4f0cad194c5742c692d7152d92cd5565: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 14:43:10.938656 7774 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 14:43:10.939481 7773 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 83383d65b4da4ba5b62b800da9c65e9f (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: false } } }
I20260619 14:43:10.940076 7309 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 83383d65b4da4ba5b62b800da9c65e9f (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: false } } }
I20260619 14:43:10.941026 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4f0cad194c5742c692d7152d92cd5565" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: false } } }
I20260619 14:43:10.941138 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:10.941224 7121 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 83383d65b4da4ba5b62b800da9c65e9f (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: false } } }
I20260619 14:43:10.941305 7267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f0cad194c5742c692d7152d92cd5565" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: false } } }
I20260619 14:43:10.941426 7267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:10.941457 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f0cad194c5742c692d7152d92cd5565" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: false } } }
I20260619 14:43:10.941551 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:10.942287 7342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f0cad194c5742c692d7152d92cd5565" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: false } } }
I20260619 14:43:10.942416 7342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:43:10.942847 7782 sys_catalog.cc:455] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f0cad194c5742c692d7152d92cd5565" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: false } } }
I20260619 14:43:10.943069 7782 sys_catalog.cc:458] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:43:10.950984 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.944787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38627 (local address 127.0.0.1:36610)
0619 14:43:10.945294 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.945312 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.945391 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.946396 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.946399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.946416 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:10.946505 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.946509 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.947805 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.947811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.948557 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.948610 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.950544 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.950548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.950633 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.950659 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.950747 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.950842 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422}
I20260619 14:43:10.951272 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.944961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.0.98.1:38627)
0619 14:43:10.945398 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.945413 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.945464 (+ 51us) server_negotiation.cc:407] Connection header received
0619 14:43:10.945552 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.945560 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.945811 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.946422 (+ 611us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:10.946631 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.947701 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.949552 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.950428 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.950537 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.950691 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.950781 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.950878 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":98,"threads_started":1}
I20260619 14:43:10.951507 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.945246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33247 (local address 127.0.0.1:41454)
0619 14:43:10.946645 (+ 1399us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.946658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.946727 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.947210 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.947213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.947232 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:10.947317 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.947323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.948690 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.948693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.949714 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.949723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.951017 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.951022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.951109 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.951140 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.951265 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.951358 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1344,"mutex_wait_us":23,"thread_start_us":106,"threads_started":1}
I20260619 14:43:10.951628 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.945636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.0.98.3:33247)
0619 14:43:10.946729 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.946740 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.946766 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:43:10.946858 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.946865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.947064 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.947210 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:10.947415 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.948562 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.949850 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.950726 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.950833 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.950955 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.951292 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.951472 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":982,"thread_start_us":98,"threads_started":1}
I20260619 14:43:10.951730 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.944994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34115 (local address 127.0.0.1:34214)
0619 14:43:10.946386 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:10.946400 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:10.946474 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:10.947500 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:10.947502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:10.947516 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:10.947585 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.947589 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.948823 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.948826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.949934 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:10.949944 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.951226 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.951230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:10.951328 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.951363 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 14:43:10.951505 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 14:43:10.951591 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1320,"mutex_wait_us":1,"thread_start_us":122,"threads_started":1}
I20260619 14:43:10.952124 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:10.945159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34214 (local address 127.0.98.2:34115)
0619 14:43:10.946966 (+ 1807us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:10.946978 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:10.946997 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:43:10.947060 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:10.947065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:10.947218 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:10.947357 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:10.947691 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.948731 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.950075 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:10.950916 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:10.951022 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:10.951165 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:10.951899 (+ 734us) server_negotiation.cc:299] Negotiation successful
0619 14:43:10.951996 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1711,"thread_start_us":106,"threads_started":1}
I20260619 14:43:10.954452 7690 tablet_service.cc:1467] Processing CreateTablet for tablet bdadc4ba3ec54df78d48d5bd3b6bb95a (DEFAULT_TABLE table=test-table [id=96b4b7214b4843239a52c543a8fed45c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:10.954459 7553 tablet_service.cc:1467] Processing CreateTablet for tablet bdadc4ba3ec54df78d48d5bd3b6bb95a (DEFAULT_TABLE table=test-table [id=96b4b7214b4843239a52c543a8fed45c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:10.954437 7416 tablet_service.cc:1467] Processing CreateTablet for tablet bdadc4ba3ec54df78d48d5bd3b6bb95a (DEFAULT_TABLE table=test-table [id=96b4b7214b4843239a52c543a8fed45c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:10.955533 7553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdadc4ba3ec54df78d48d5bd3b6bb95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:10.955533 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdadc4ba3ec54df78d48d5bd3b6bb95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:10.955541 7416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdadc4ba3ec54df78d48d5bd3b6bb95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:10.962451 7788 tablet_bootstrap.cc:492] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143: Bootstrap starting.
I20260619 14:43:10.965097 7788 tablet_bootstrap.cc:654] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:10.966002 7788 log.cc:824] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:10.966717 7789 tablet_bootstrap.cc:492] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32: Bootstrap starting.
I20260619 14:43:10.967797 7790 tablet_bootstrap.cc:492] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a: Bootstrap starting.
I20260619 14:43:10.968070 7788 tablet_bootstrap.cc:492] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143: No bootstrap required, opened a new log
I20260619 14:43:10.968262 7788 ts_tablet_manager.cc:1397] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:43:10.969887 7790 tablet_bootstrap.cc:654] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:10.970677 7790 log.cc:824] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:10.971997 7788 raft_consensus.cc:348] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } } peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } }
I20260619 14:43:10.972229 7788 raft_consensus.cc:374] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:10.972302 7788 raft_consensus.cc:729] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2f8ba8faff4d7dab0982c9b75bf143, State: Initialized, Role: FOLLOWER
I20260619 14:43:10.972472 7789 tablet_bootstrap.cc:654] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:10.972740 7788 consensus_queue.cc:260] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } } peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } }
I20260619 14:43:10.973248 7789 log.cc:824] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:10.973379 7483 heartbeater.cc:503] Master 127.0.98.62:35593 was elected leader, sending a full tablet report...
I20260619 14:43:10.973707 7788 ts_tablet_manager.cc:1428] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260619 14:43:10.975306 7790 tablet_bootstrap.cc:492] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a: No bootstrap required, opened a new log
I20260619 14:43:10.975541 7790 ts_tablet_manager.cc:1397] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 14:43:10.980875 7790 raft_consensus.cc:348] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } } peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } }
I20260619 14:43:10.981135 7790 raft_consensus.cc:374] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:10.981176 7790 raft_consensus.cc:729] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ad98e2a58d4320aaf58238a845e40a, State: Initialized, Role: FOLLOWER
I20260619 14:43:10.981622 7790 consensus_queue.cc:260] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } } peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } }
I20260619 14:43:10.983088 7620 heartbeater.cc:503] Master 127.0.98.62:35593 was elected leader, sending a full tablet report...
I20260619 14:43:10.985246 7790 ts_tablet_manager.cc:1428] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260619 14:43:10.986593 7789 tablet_bootstrap.cc:492] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32: No bootstrap required, opened a new log
I20260619 14:43:10.986814 7789 ts_tablet_manager.cc:1397] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20260619 14:43:10.990666 7789 raft_consensus.cc:348] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } } peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } }
I20260619 14:43:10.990942 7789 raft_consensus.cc:374] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:10.990988 7789 raft_consensus.cc:729] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875fc9e2d4544467a24fce99d075ed32, State: Initialized, Role: FOLLOWER
I20260619 14:43:10.991406 7789 consensus_queue.cc:260] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } } peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } }
I20260619 14:43:10.992166 7757 heartbeater.cc:503] Master 127.0.98.62:35593 was elected leader, sending a full tablet report...
I20260619 14:43:10.992491 7789 ts_tablet_manager.cc:1428] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:43:11.081625 7793 raft_consensus.cc:482] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:43:11.081892 7793 raft_consensus.cc:504] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } } peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } }
I20260619 14:43:11.083209 7793 leader_election.cc:288] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ad98e2a58d4320aaf58238a845e40a (127.0.98.2:34115), 875fc9e2d4544467a24fce99d075ed32 (127.0.98.3:33247)
I20260619 14:43:11.087054 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:11.083863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39875 (local address 127.0.98.2:34115)
0619 14:43:11.083998 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:11.084004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:11.084018 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:43:11.084072 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:11.084074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:11.084112 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:11.084207 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:11.084455 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:11.085000 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:11.086123 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:11.086509 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:11.086622 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:11.086685 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:11.086853 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 14:43:11.086919 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:43:11.087100 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:11.083613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43761 (local address 127.0.98.3:33247)
0619 14:43:11.083742 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:11.083746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:11.083895 (+ 149us) server_negotiation.cc:407] Connection header received
0619 14:43:11.084056 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:11.084059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:11.084111 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:11.084195 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:11.084503 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:11.085000 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:11.086122 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:11.086555 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:11.086673 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:11.086746 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:11.086912 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 14:43:11.086976 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:43:11.087105 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:11.083434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34115 (local address 127.0.98.1:39875)
0619 14:43:11.083788 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:11.083806 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:11.083918 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:11.084214 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:11.084217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:11.084233 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:11.084347 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:11.084353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:11.085109 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:11.085114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:11.085996 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:11.086005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:11.086626 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:11.086629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:11.086734 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:11.086753 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:43:11.086839 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 14:43:11.086909 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":47,"thread_start_us":77,"threads_started":1}
I20260619 14:43:11.087132 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:11.083540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33247 (local address 127.0.98.1:43761)
0619 14:43:11.083783 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:11.083799 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:11.083937 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:11.084207 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:11.084210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:11.084224 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:11.084378 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:11.084384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:11.085114 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:11.085118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:11.085995 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:11.086004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:11.086676 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:11.086680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:11.086779 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:11.086800 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:43:11.086897 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 14:43:11.086968 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":65,"thread_start_us":81,"threads_started":1}
I20260619 14:43:11.087953 7573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdadc4ba3ec54df78d48d5bd3b6bb95a" candidate_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ad98e2a58d4320aaf58238a845e40a" is_pre_election: true
I20260619 14:43:11.087960 7710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdadc4ba3ec54df78d48d5bd3b6bb95a" candidate_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875fc9e2d4544467a24fce99d075ed32" is_pre_election: true
I20260619 14:43:11.088246 7573 raft_consensus.cc:2454] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf2f8ba8faff4d7dab0982c9b75bf143 in term 0.
I20260619 14:43:11.088253 7710 raft_consensus.cc:2454] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf2f8ba8faff4d7dab0982c9b75bf143 in term 0.
I20260619 14:43:11.088680 7371 leader_election.cc:302] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ad98e2a58d4320aaf58238a845e40a, bf2f8ba8faff4d7dab0982c9b75bf143; no voters:
I20260619 14:43:11.088968 7793 raft_consensus.cc:2781] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:43:11.089048 7793 raft_consensus.cc:482] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:43:11.089093 7793 raft_consensus.cc:3037] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:11.090550 7793 raft_consensus.cc:504] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } } peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } }
I20260619 14:43:11.090915 7793 leader_election.cc:288] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [CANDIDATE]: Term 1 election: Requested vote from peers 23ad98e2a58d4320aaf58238a845e40a (127.0.98.2:34115), 875fc9e2d4544467a24fce99d075ed32 (127.0.98.3:33247)
I20260619 14:43:11.091364 7710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdadc4ba3ec54df78d48d5bd3b6bb95a" candidate_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875fc9e2d4544467a24fce99d075ed32"
I20260619 14:43:11.091521 7710 raft_consensus.cc:3037] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:11.091626 7573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdadc4ba3ec54df78d48d5bd3b6bb95a" candidate_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ad98e2a58d4320aaf58238a845e40a"
I20260619 14:43:11.091747 7573 raft_consensus.cc:3037] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:11.092896 7710 raft_consensus.cc:2454] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf2f8ba8faff4d7dab0982c9b75bf143 in term 1.
I20260619 14:43:11.092978 7573 raft_consensus.cc:2454] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf2f8ba8faff4d7dab0982c9b75bf143 in term 1.
I20260619 14:43:11.093235 7372 leader_election.cc:302] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875fc9e2d4544467a24fce99d075ed32, bf2f8ba8faff4d7dab0982c9b75bf143; no voters:
I20260619 14:43:11.093394 7793 raft_consensus.cc:2781] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:11.093611 7793 raft_consensus.cc:686] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [term 1 LEADER]: Becoming Leader. State: Replica: bf2f8ba8faff4d7dab0982c9b75bf143, State: Running, Role: LEADER
I20260619 14:43:11.093959 7793 consensus_queue.cc:237] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } } peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } }
I20260619 14:43:11.097301 6518 catalog_manager.cc:5495] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf2f8ba8faff4d7dab0982c9b75bf143 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } health_report { overall_health: UNKNOWN } } }
I20260619 14:43:11.138542 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:11.134872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36626 (local address 127.0.98.1:38627)
0619 14:43:11.135006 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:11.135009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:11.135025 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:43:11.135100 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:11.135102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:11.135157 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:11.135253 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:43:11.135604 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:11.136139 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:11.137062 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:11.137278 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:11.137375 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:11.137639 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:43:11.138118 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:43:11.138223 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:11.138361 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 14:43:11.138422 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:43:11.144999 7573 raft_consensus.cc:1261] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a [term 1 FOLLOWER]: Refusing update from remote peer bf2f8ba8faff4d7dab0982c9b75bf143: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:11.145054 7710 raft_consensus.cc:1261] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32 [term 1 FOLLOWER]: Refusing update from remote peer bf2f8ba8faff4d7dab0982c9b75bf143: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:11.145749 7793 consensus_queue.cc:1035] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:11.145968 7799 consensus_queue.cc:1035] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:43:11.148274 7759 tablet.cc:2369] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:43:11.155474 7804 mvcc.cc:205] Tried to move back new op lower bound from 7298581262922215424 to 7298581262723530752. Current Snapshot: MvccSnapshot[applied={T|T < 7298581262922215424}]
I20260619 14:43:11.155818 7802 mvcc.cc:205] Tried to move back new op lower bound from 7298581262922215424 to 7298581262723530752. Current Snapshot: MvccSnapshot[applied={T|T < 7298581262922215424}]
I20260619 14:43:11.161614 7803 mvcc.cc:205] Tried to move back new op lower bound from 7298581262922215424 to 7298581262723530752. Current Snapshot: MvccSnapshot[applied={T|T < 7298581262922215424 or (T in {7298581262922215424})}]
I20260619 14:43:11.164793 6518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:43:11.164938 6518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:43:11.169158 6518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f0cad194c5742c692d7152d92cd5565: Sending DeleteTablet for 3 replicas of tablet bdadc4ba3ec54df78d48d5bd3b6bb95a
I20260619 14:43:11.170341 7416 tablet_service.cc:1514] Processing DeleteTablet for tablet bdadc4ba3ec54df78d48d5bd3b6bb95a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36610
I20260619 14:43:11.170320 7553 tablet_service.cc:1514] Processing DeleteTablet for tablet bdadc4ba3ec54df78d48d5bd3b6bb95a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34214
I20260619 14:43:11.171376 7690 tablet_service.cc:1514] Processing DeleteTablet for tablet bdadc4ba3ec54df78d48d5bd3b6bb95a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41454
I20260619 14:43:11.171941 7814 tablet_replica.cc:330] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32: stopping tablet replica
I20260619 14:43:11.173730 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 6488
I20260619 14:43:11.174530 7814 raft_consensus.cc:2229] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:43:11.175513 7814 raft_consensus.cc:2258] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:11.177831 7814 ts_tablet_manager.cc:1905] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:11.181221 7814 ts_tablet_manager.cc:1918] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:43:11.181357 7814 log.cc:1196] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/bdadc4ba3ec54df78d48d5bd3b6bb95a
I20260619 14:43:11.181489 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:11.171807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.0.98.61:34175)
0619 14:43:11.171950 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:11.171956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:11.171971 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:43:11.172009 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:11.172012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:11.172059 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:11.172213 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:11.173104 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:11.173676 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:11.174765 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:11.175064 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:11.175189 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:11.177003 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:11.177006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:11.177009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:11.177037 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:11.178837 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:11.179390 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:11.179394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:11.179396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:11.179502 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:11.179822 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:11.179827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:11.179829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:11.180059 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:11.180205 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:11.181045 (+ 840us) server_negotiation.cc:299] Negotiation successful
0619 14:43:11.181296 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:43:11.181770 7814 ts_tablet_manager.cc:1939] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 875fc9e2d4544467a24fce99d075ed32: Deleting consensus metadata
W20260619 14:43:11.184096 7645 connection.cc:535] server connection from 127.0.0.1:41454 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:43:11.187623 7812 tablet_replica.cc:330] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143: stopping tablet replica
I20260619 14:43:11.188050 7812 raft_consensus.cc:2229] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:43:11.188514 7812 raft_consensus.cc:2258] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:11.190356 7812 ts_tablet_manager.cc:1905] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:11.190869 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:11.171910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38894 (local address 127.0.98.60:44173)
0619 14:43:11.172018 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:11.172023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:11.180935 (+ 8912us) server_negotiation.cc:407] Connection header received
0619 14:43:11.181873 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:11.181880 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:11.181962 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:11.182025 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:11.183006 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:11.183770 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:11.184628 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:11.184976 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:11.185079 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:11.186879 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:11.186883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:11.186886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:11.186919 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:11.188963 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:11.189466 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:11.189470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:11.189472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:11.189559 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:11.189899 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:11.189905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:11.189909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:11.190132 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:11.190239 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:11.190584 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 14:43:11.190702 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:43:11.193444 7812 ts_tablet_manager.cc:1918] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:43:11.193740 7811 tablet_replica.cc:330] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a: stopping tablet replica
I20260619 14:43:11.194164 7811 raft_consensus.cc:2229] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:43:11.194581 7811 raft_consensus.cc:2258] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:11.194720 7812 log.cc:1196] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-0/wal/wals/bdadc4ba3ec54df78d48d5bd3b6bb95a
I20260619 14:43:11.195111 7812 ts_tablet_manager.cc:1939] T bdadc4ba3ec54df78d48d5bd3b6bb95a P bf2f8ba8faff4d7dab0982c9b75bf143: Deleting consensus metadata
W20260619 14:43:11.196527 7369 connection.cc:441] server connection from 127.0.0.1:36610 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36610 (request call id 1) could send its response
I20260619 14:43:11.198050 7811 ts_tablet_manager.cc:1905] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:11.199393 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:11.187953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.0.98.60:44173)
0619 14:43:11.188071 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:11.188075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:11.189055 (+ 980us) server_negotiation.cc:407] Connection header received
0619 14:43:11.189230 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:11.189233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:11.189289 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:11.189343 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:11.190228 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:11.190954 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:11.192726 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:11.193078 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:11.193155 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:11.195034 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:11.195038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:11.195041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:11.195065 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:11.196813 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:11.197297 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:11.197301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:11.197302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:11.197385 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:11.198032 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:11.198038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:11.198041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:11.198294 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:11.198416 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:11.198776 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 14:43:11.198904 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:43:11.201143 7811 ts_tablet_manager.cc:1918] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:43:11.201306 7811 log.cc:1196] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/bdadc4ba3ec54df78d48d5bd3b6bb95a
I20260619 14:43:11.201623 7811 ts_tablet_manager.cc:1939] T bdadc4ba3ec54df78d48d5bd3b6bb95a P 23ad98e2a58d4320aaf58238a845e40a: Deleting consensus metadata
I20260619 14:43:11.202230 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:11.188465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48432 (local address 127.0.98.61:34175)
0619 14:43:11.188582 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:11.188586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:11.192671 (+ 4085us) server_negotiation.cc:407] Connection header received
0619 14:43:11.192846 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:11.192849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:11.192897 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:11.192946 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:11.194515 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:11.195327 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:11.196631 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:11.196999 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:11.197078 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:11.198742 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:11.198746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:11.198749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:11.198775 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:11.200498 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:11.200938 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:11.200941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:11.200943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:11.201027 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:11.201313 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:11.201317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:11.201319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:11.201524 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:11.201692 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:11.201953 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 14:43:11.202070 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 14:43:11.202765 7507 connection.cc:441] server connection from 127.0.0.1:34214 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34214 (request call id 1) could send its response
I20260619 14:43:12.157971 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.157410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.2:51471)
0619 14:43:12.157749 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.157873 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":89,"threads_started":1}
W20260619 14:43:12.158382 7620 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35593 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
I20260619 14:43:12.162793 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.162299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.3:55735)
0619 14:43:12.162575 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.162684 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
W20260619 14:43:12.163478 7757 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35593 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
I20260619 14:43:12.163929 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.163114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.1:47695)
0619 14:43:12.163764 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.163847 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":60,"threads_started":1}
W20260619 14:43:12.164242 7483 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35593 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
I20260619 14:43:12.670137 7825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4f0cad194c5742c692d7152d92cd5565)
I20260619 14:43:12.670392 7825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: false } }
I20260619 14:43:12.670661 7826 raft_consensus.cc:482] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4f0cad194c5742c692d7152d92cd5565)
I20260619 14:43:12.670859 7826 raft_consensus.cc:504] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: false } }
I20260619 14:43:12.671648 7825 leader_election.cc:288] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f0cad194c5742c692d7152d92cd5565 (127.0.98.62:35593), a918948481984a82bef65cbedfad5ef7 (127.0.98.61:34175)
I20260619 14:43:12.672075 7826 leader_election.cc:288] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f0cad194c5742c692d7152d92cd5565 (127.0.98.62:35593), 83383d65b4da4ba5b62b800da9c65e9f (127.0.98.60:44173)
I20260619 14:43:12.672415 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.671898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.0.1:42998)
0619 14:43:12.672230 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.672312 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":58,"threads_started":1}
W20260619 14:43:12.672576 7107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
I20260619 14:43:12.672799 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.671730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.0.1:42992)
0619 14:43:12.672543 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.672702 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":85,"threads_started":1}
W20260619 14:43:12.672968 7290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
I20260619 14:43:12.674081 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.673782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.0.1:43020)
0619 14:43:12.673977 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.674020 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260619 14:43:12.674268 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.674004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.0.1:43026)
0619 14:43:12.674152 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.674201 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260619 14:43:12.674422 7290 leader_election.cc:334] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4f0cad194c5742c692d7152d92cd5565 (127.0.98.62:35593): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
W20260619 14:43:12.674477 7107 leader_election.cc:334] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4f0cad194c5742c692d7152d92cd5565 (127.0.98.62:35593): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
I20260619 14:43:12.676868 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.671940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44173 (local address 127.0.0.1:38920)
0619 14:43:12.672233 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.672247 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:12.672344 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:12.672803 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:12.672809 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:12.672832 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:12.673167 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:12.673181 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.674462 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.674466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:12.675790 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:12.675796 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.676458 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.676461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:12.676550 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:12.676585 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 14:43:12.676673 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 14:43:12.676746 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":42,"threads_started":1}
I20260619 14:43:12.676882 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.672103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38920 (local address 127.0.98.60:44173)
0619 14:43:12.672393 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:12.672399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:12.672418 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:43:12.672570 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:12.672574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:12.672627 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:12.673293 (+ 666us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:12.673404 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.674251 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.675898 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.676329 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.676449 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:12.676509 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:12.676694 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 14:43:12.676765 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":54,"threads_started":1}
I20260619 14:43:12.677405 7309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a918948481984a82bef65cbedfad5ef7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "83383d65b4da4ba5b62b800da9c65e9f" is_pre_election: true
I20260619 14:43:12.677604 7309 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a918948481984a82bef65cbedfad5ef7 in term 1.
I20260619 14:43:12.677963 7106 leader_election.cc:302] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83383d65b4da4ba5b62b800da9c65e9f, a918948481984a82bef65cbedfad5ef7; no voters: 4f0cad194c5742c692d7152d92cd5565
I20260619 14:43:12.678157 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.672363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34175 (local address 127.0.0.1:48440)
0619 14:43:12.672753 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.672767 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:12.674291 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:12.674666 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:12.674668 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:12.674692 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:12.674759 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:12.674762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.675749 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.675752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:12.676622 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:12.676697 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.677715 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.677718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:12.677819 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:12.677843 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:43:12.677939 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 14:43:12.678012 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":57,"threads_started":1}
I20260619 14:43:12.678156 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.672157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.0.98.61:34175)
0619 14:43:12.672627 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:12.672632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:12.672829 (+ 197us) server_negotiation.cc:407] Connection header received
0619 14:43:12.674422 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:12.674425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:12.674468 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:12.674525 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:12.674869 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.675607 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.676843 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.677592 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.677716 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:12.677771 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:12.677954 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 14:43:12.678022 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":408,"thread_start_us":76,"threads_started":1}
I20260619 14:43:12.678180 7826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 14:43:12.678244 7826 raft_consensus.cc:482] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4f0cad194c5742c692d7152d92cd5565)
I20260619 14:43:12.678284 7826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:43:12.678718 7121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "83383d65b4da4ba5b62b800da9c65e9f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a918948481984a82bef65cbedfad5ef7" is_pre_election: true
I20260619 14:43:12.679563 7826 raft_consensus.cc:504] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: false } }
I20260619 14:43:12.679818 7121 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83383d65b4da4ba5b62b800da9c65e9f in current term 2: Already voted for candidate a918948481984a82bef65cbedfad5ef7 in this term.
I20260619 14:43:12.679944 7826 leader_election.cc:288] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [CANDIDATE]: Term 2 election: Requested vote from peers 4f0cad194c5742c692d7152d92cd5565 (127.0.98.62:35593), 83383d65b4da4ba5b62b800da9c65e9f (127.0.98.60:44173)
I20260619 14:43:12.680296 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.680050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.0.1:43032)
0619 14:43:12.680134 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.680187 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260619 14:43:12.680274 7309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a918948481984a82bef65cbedfad5ef7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "83383d65b4da4ba5b62b800da9c65e9f"
I20260619 14:43:12.680279 7290 leader_election.cc:302] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83383d65b4da4ba5b62b800da9c65e9f; no voters: 4f0cad194c5742c692d7152d92cd5565, a918948481984a82bef65cbedfad5ef7
I20260619 14:43:12.680397 7309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:43:12.681463 7309 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a918948481984a82bef65cbedfad5ef7 in term 2.
I20260619 14:43:12.681566 7825 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:43:12.681798 7106 leader_election.cc:302] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83383d65b4da4ba5b62b800da9c65e9f, a918948481984a82bef65cbedfad5ef7; no voters:
I20260619 14:43:12.682029 7826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:43:12.682281 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.682022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.0.1:43046)
0619 14:43:12.682155 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.682205 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 14:43:12.682293 7826 raft_consensus.cc:686] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [term 2 LEADER]: Becoming Leader. State: Replica: a918948481984a82bef65cbedfad5ef7, State: Running, Role: LEADER
I20260619 14:43:12.682588 7826 consensus_queue.cc:237] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: false } }
W20260619 14:43:12.683524 7107 leader_election.cc:334] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4f0cad194c5742c692d7152d92cd5565 (127.0.98.62:35593): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
I20260619 14:43:12.683805 7835 sys_catalog.cc:455] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a918948481984a82bef65cbedfad5ef7. Latest consensus state: current_term: 2 leader_uuid: "a918948481984a82bef65cbedfad5ef7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: false } } }
I20260619 14:43:12.683919 7835 sys_catalog.cc:458] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:12.684242 7837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:43:12.686445 7837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:43:12.686846 7837 catalog_manager.cc:1259] Loaded cluster ID: c48d93b0efcf4a02a9e36ec812ca32d4
I20260619 14:43:12.686892 7837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:43:12.687479 7837 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:43:12.688174 7837 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: Loaded TSK: 0
I20260619 14:43:12.688952 7837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:43:12.734297 7117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:43:12.737475 7117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:43:12.740410 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.740143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.0.1:43064)
0619 14:43:12.740269 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.740330 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 14:43:12.740516 7309 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [term 2 FOLLOWER]: Refusing update from remote peer a918948481984a82bef65cbedfad5ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 14:43:12.740988 7826 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 14:43:12.741125 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.740942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.0.1:43068)
0619 14:43:12.741030 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.741073 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260619 14:43:12.741241 7107 consensus_peers.cc:487] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 -> Peer 4f0cad194c5742c692d7152d92cd5565 (127.0.98.62:35593): Couldn't send request to peer 4f0cad194c5742c692d7152d92cd5565. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 14:43:12.742563 7825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a918948481984a82bef65cbedfad5ef7. Latest consensus state: current_term: 2 leader_uuid: "a918948481984a82bef65cbedfad5ef7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: false } } }
I20260619 14:43:12.742698 7825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:43:12.743906 7826 sys_catalog.cc:455] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a918948481984a82bef65cbedfad5ef7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: false } } }
I20260619 14:43:12.744032 7826 sys_catalog.cc:458] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:12.744293 7825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a918948481984a82bef65cbedfad5ef7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4f0cad194c5742c692d7152d92cd5565" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35593 } } peers { permanent_uuid: "a918948481984a82bef65cbedfad5ef7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34175 } attrs { promote: false } } peers { permanent_uuid: "83383d65b4da4ba5b62b800da9c65e9f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44173 } attrs { promote: false } } }
I20260619 14:43:12.744450 7825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83383d65b4da4ba5b62b800da9c65e9f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:43:12.754142 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.749911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38627 (local address 127.0.0.1:36640)
0619 14:43:12.749998 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.750010 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:12.750132 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:12.750677 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:12.750680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:12.750698 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:12.750947 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:12.750953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.752060 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.752063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:12.752992 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:12.753000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.753702 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.753705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:12.753818 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:12.753847 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:43:12.753921 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:43:12.753994 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":117,"spinlock_wait_cycles":7552}
I20260619 14:43:12.754196 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.750202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36640 (local address 127.0.98.1:38627)
0619 14:43:12.750451 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:12.750457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:12.750476 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:43:12.750526 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:12.750528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:12.750565 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:12.750659 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:12.751072 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.751917 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.753115 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.753566 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.753691 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:12.753763 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:12.753951 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 14:43:12.754026 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":118,"threads_started":1}
I20260619 14:43:12.755139 7416 tablet_service.cc:1467] Processing CreateTablet for tablet 7b5c4bddfb5740cca5de8280c1f3c4d1 (DEFAULT_TABLE table=test-table [id=d6f7dfca771048c6a71f90b7381e1482]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:12.755527 7416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b5c4bddfb5740cca5de8280c1f3c4d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:12.755795 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.751128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34115 (local address 127.0.0.1:34220)
0619 14:43:12.751364 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.751378 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:12.751484 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:12.751949 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:12.751952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:12.751967 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:12.752098 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:12.752105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.752808 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.752818 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:12.754073 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:12.754086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.755325 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.755329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:12.755427 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:12.755476 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 14:43:12.755561 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 14:43:12.755633 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":4,"thread_start_us":109,"threads_started":1}
I20260619 14:43:12.756076 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.749709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33247 (local address 127.0.0.1:41456)
0619 14:43:12.749806 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.749821 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:12.749902 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:12.750696 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:12.750699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:12.750720 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:12.750913 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:12.750920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.752052 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.752056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:12.754780 (+ 2724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:12.754789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.755761 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.755765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:12.755853 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:12.755878 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:43:12.755930 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 14:43:12.755998 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":1594}
I20260619 14:43:12.756093 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.751431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34220 (local address 127.0.98.2:34115)
0619 14:43:12.751745 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:12.751749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:12.751760 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:43:12.751800 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:12.751802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:12.751843 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:12.751942 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:12.752205 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.752702 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.754538 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.755188 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.755368 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:12.755473 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:12.755666 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 14:43:12.755747 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":70,"threads_started":1}
I20260619 14:43:12.756652 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.750191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41456 (local address 127.0.98.3:33247)
0619 14:43:12.750441 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:12.750446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:12.750461 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:43:12.750515 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:12.750518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:12.750569 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:12.750809 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:12.751085 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.751917 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.754923 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.755620 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.756368 (+ 748us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:12.756429 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:12.756467 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 14:43:12.756542 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":111,"threads_started":1}
I20260619 14:43:12.757184 7690 tablet_service.cc:1467] Processing CreateTablet for tablet 7b5c4bddfb5740cca5de8280c1f3c4d1 (DEFAULT_TABLE table=test-table [id=d6f7dfca771048c6a71f90b7381e1482]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:12.757457 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b5c4bddfb5740cca5de8280c1f3c4d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:12.757972 7848 tablet_bootstrap.cc:492] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P bf2f8ba8faff4d7dab0982c9b75bf143: Bootstrap starting.
I20260619 14:43:12.758798 7553 tablet_service.cc:1467] Processing CreateTablet for tablet 7b5c4bddfb5740cca5de8280c1f3c4d1 (DEFAULT_TABLE table=test-table [id=d6f7dfca771048c6a71f90b7381e1482]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:12.759017 7553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b5c4bddfb5740cca5de8280c1f3c4d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:12.759842 7849 tablet_bootstrap.cc:492] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32: Bootstrap starting.
I20260619 14:43:12.759924 7848 tablet_bootstrap.cc:654] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P bf2f8ba8faff4d7dab0982c9b75bf143: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:12.760841 7849 tablet_bootstrap.cc:654] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:12.761088 7848 tablet_bootstrap.cc:492] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P bf2f8ba8faff4d7dab0982c9b75bf143: No bootstrap required, opened a new log
I20260619 14:43:12.761382 7848 ts_tablet_manager.cc:1397] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P bf2f8ba8faff4d7dab0982c9b75bf143: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:43:12.761523 7851 tablet_bootstrap.cc:492] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a: Bootstrap starting.
I20260619 14:43:12.761847 7848 raft_consensus.cc:348] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P bf2f8ba8faff4d7dab0982c9b75bf143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } } peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } }
I20260619 14:43:12.761991 7848 raft_consensus.cc:374] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P bf2f8ba8faff4d7dab0982c9b75bf143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:12.762030 7848 raft_consensus.cc:729] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P bf2f8ba8faff4d7dab0982c9b75bf143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2f8ba8faff4d7dab0982c9b75bf143, State: Initialized, Role: FOLLOWER
I20260619 14:43:12.762166 7848 consensus_queue.cc:260] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P bf2f8ba8faff4d7dab0982c9b75bf143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } } peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } }
I20260619 14:43:12.762519 7848 ts_tablet_manager.cc:1428] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P bf2f8ba8faff4d7dab0982c9b75bf143: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:43:12.762598 7851 tablet_bootstrap.cc:654] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:12.762893 7849 tablet_bootstrap.cc:492] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32: No bootstrap required, opened a new log
I20260619 14:43:12.762988 7849 ts_tablet_manager.cc:1397] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:43:12.763484 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.763032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.1:50443)
0619 14:43:12.763283 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.763351 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":100,"threads_started":1}
I20260619 14:43:12.763507 7849 raft_consensus.cc:348] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } } peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } }
I20260619 14:43:12.763628 7849 raft_consensus.cc:374] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:12.763660 7849 raft_consensus.cc:729] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875fc9e2d4544467a24fce99d075ed32, State: Initialized, Role: FOLLOWER
I20260619 14:43:12.763805 7849 consensus_queue.cc:260] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } } peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } }
I20260619 14:43:12.764278 7849 ts_tablet_manager.cc:1428] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:43:12.765265 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.764423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.3:57045)
0619 14:43:12.765100 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.765161 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":94,"threads_started":1}
I20260619 14:43:12.766161 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.765706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.3:43315)
0619 14:43:12.766041 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.766095 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261}
I20260619 14:43:12.766506 7856 raft_consensus.cc:482] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:43:12.766650 7856 raft_consensus.cc:504] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } } peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } }
I20260619 14:43:12.767011 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.766718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.1:54971)
0619 14:43:12.766878 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.766926 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 14:43:12.767360 7484 heartbeater.cc:503] Master 127.0.98.61:34175 was elected leader, sending a full tablet report...
I20260619 14:43:12.767977 7856 leader_election.cc:288] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf2f8ba8faff4d7dab0982c9b75bf143 (127.0.98.1:38627), 23ad98e2a58d4320aaf58238a845e40a (127.0.98.2:34115)
I20260619 14:43:12.768206 7851 tablet_bootstrap.cc:492] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a: No bootstrap required, opened a new log
I20260619 14:43:12.768303 7851 ts_tablet_manager.cc:1397] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260619 14:43:12.768798 7851 raft_consensus.cc:348] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } } peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } }
I20260619 14:43:12.768906 7851 raft_consensus.cc:374] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:12.768940 7851 raft_consensus.cc:729] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ad98e2a58d4320aaf58238a845e40a, State: Initialized, Role: FOLLOWER
I20260619 14:43:12.769068 7851 consensus_queue.cc:260] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } } peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } }
I20260619 14:43:12.769492 7851 ts_tablet_manager.cc:1428] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 14:43:12.771596 7621 heartbeater.cc:503] Master 127.0.98.61:34175 was elected leader, sending a full tablet report...
I20260619 14:43:12.772186 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.769971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.2:50473)
0619 14:43:12.772031 (+ 2060us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.772086 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1998,"thread_start_us":99,"threads_started":1}
I20260619 14:43:12.773010 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.768718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42109 (local address 127.0.98.2:34115)
0619 14:43:12.768831 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:12.768836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:12.768850 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:43:12.768893 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:12.768897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:12.768945 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:12.769027 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:12.769447 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.770141 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.771720 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.772324 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.772400 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:12.772462 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:12.772827 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 14:43:12.772913 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 14:43:12.773068 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.768711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58473 (local address 127.0.98.1:38627)
0619 14:43:12.768813 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:12.768816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:12.768825 (+ 9us) server_negotiation.cc:407] Connection header received
0619 14:43:12.768858 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:12.768861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:12.768894 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:12.768980 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:12.769270 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.770061 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.771863 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.772443 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.772514 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:12.772565 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:12.772899 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 14:43:12.772965 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:43:12.773419 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.768113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38627 (local address 127.0.98.3:58473)
0619 14:43:12.768229 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.768244 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:12.768345 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:12.768991 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:12.768994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:12.769010 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:12.769116 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:12.769123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.770198 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.770204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:12.771733 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:12.771741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.772641 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.772644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:12.772765 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:12.772795 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:43:12.773170 (+ 375us) client_negotiation.cc:240] Negotiation successful
0619 14:43:12.773287 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":198}
I20260619 14:43:12.773422 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.768343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34115 (local address 127.0.98.3:42109)
0619 14:43:12.768567 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.768582 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:12.768664 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:12.769204 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:12.769207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:12.769226 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:12.769321 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:12.769327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.770313 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.770316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:12.771592 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:12.771602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.772555 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.772558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:12.772691 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:12.772713 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:43:12.773224 (+ 511us) client_negotiation.cc:240] Negotiation successful
0619 14:43:12.773307 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":87,"thread_start_us":99,"threads_started":1}
I20260619 14:43:12.773475 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.772897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.2:51937)
0619 14:43:12.773362 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.773411 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395}
I20260619 14:43:12.773857 7573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5c4bddfb5740cca5de8280c1f3c4d1" candidate_uuid: "875fc9e2d4544467a24fce99d075ed32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ad98e2a58d4320aaf58238a845e40a" is_pre_election: true
I20260619 14:43:12.773958 7573 raft_consensus.cc:2454] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 875fc9e2d4544467a24fce99d075ed32 in term 0.
I20260619 14:43:12.773988 7436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5c4bddfb5740cca5de8280c1f3c4d1" candidate_uuid: "875fc9e2d4544467a24fce99d075ed32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" is_pre_election: true
I20260619 14:43:12.774199 7436 raft_consensus.cc:2454] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P bf2f8ba8faff4d7dab0982c9b75bf143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 875fc9e2d4544467a24fce99d075ed32 in term 0.
I20260619 14:43:12.774299 7645 leader_election.cc:302] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ad98e2a58d4320aaf58238a845e40a, 875fc9e2d4544467a24fce99d075ed32; no voters:
I20260619 14:43:12.774626 7856 raft_consensus.cc:2781] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:43:12.774678 7856 raft_consensus.cc:482] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:43:12.774709 7856 raft_consensus.cc:3037] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:12.775779 7856 raft_consensus.cc:504] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } } peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } }
I20260619 14:43:12.776042 7856 leader_election.cc:288] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [CANDIDATE]: Term 1 election: Requested vote from peers bf2f8ba8faff4d7dab0982c9b75bf143 (127.0.98.1:38627), 23ad98e2a58d4320aaf58238a845e40a (127.0.98.2:34115)
I20260619 14:43:12.776389 7436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5c4bddfb5740cca5de8280c1f3c4d1" candidate_uuid: "875fc9e2d4544467a24fce99d075ed32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143"
I20260619 14:43:12.776484 7436 raft_consensus.cc:3037] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P bf2f8ba8faff4d7dab0982c9b75bf143 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:12.776435 7573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b5c4bddfb5740cca5de8280c1f3c4d1" candidate_uuid: "875fc9e2d4544467a24fce99d075ed32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ad98e2a58d4320aaf58238a845e40a"
I20260619 14:43:12.776615 7573 raft_consensus.cc:3037] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:12.777472 7436 raft_consensus.cc:2454] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P bf2f8ba8faff4d7dab0982c9b75bf143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875fc9e2d4544467a24fce99d075ed32 in term 1.
I20260619 14:43:12.777791 7645 leader_election.cc:302] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875fc9e2d4544467a24fce99d075ed32, bf2f8ba8faff4d7dab0982c9b75bf143; no voters:
I20260619 14:43:12.777803 7573 raft_consensus.cc:2454] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875fc9e2d4544467a24fce99d075ed32 in term 1.
I20260619 14:43:12.778003 7856 raft_consensus.cc:2781] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:12.778237 7856 raft_consensus.cc:686] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [term 1 LEADER]: Becoming Leader. State: Replica: 875fc9e2d4544467a24fce99d075ed32, State: Running, Role: LEADER
I20260619 14:43:12.778431 7758 heartbeater.cc:503] Master 127.0.98.61:34175 was elected leader, sending a full tablet report...
I20260619 14:43:12.778749 7856 consensus_queue.cc:237] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } } peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } }
I20260619 14:43:12.779186 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.778930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.3:47985)
0619 14:43:12.779043 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.779097 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 14:43:12.781345 7117 catalog_manager.cc:5495] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 875fc9e2d4544467a24fce99d075ed32 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "875fc9e2d4544467a24fce99d075ed32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875fc9e2d4544467a24fce99d075ed32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } health_report { overall_health: UNKNOWN } } }
I20260619 14:43:12.801894 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.797964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41464 (local address 127.0.98.3:33247)
0619 14:43:12.798114 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:12.798122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:12.798152 (+ 30us) server_negotiation.cc:407] Connection header received
0619 14:43:12.798267 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:12.798269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:12.798330 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:12.798451 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:43:12.798729 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.799306 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.800335 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:12.800578 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:12.800700 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:12.801001 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:43:12.801461 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:43:12.801600 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:12.801666 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 14:43:12.801729 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:43:12.804852 7573 raft_consensus.cc:1261] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a [term 1 FOLLOWER]: Refusing update from remote peer 875fc9e2d4544467a24fce99d075ed32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:12.805315 7436 raft_consensus.cc:1261] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P bf2f8ba8faff4d7dab0982c9b75bf143 [term 1 FOLLOWER]: Refusing update from remote peer 875fc9e2d4544467a24fce99d075ed32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:12.805424 7856 consensus_queue.cc:1035] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ad98e2a58d4320aaf58238a845e40a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:12.805881 7861 consensus_queue.cc:1035] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2f8ba8faff4d7dab0982c9b75bf143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:12.807263 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.806966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.2:43007)
0619 14:43:12.807134 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.807192 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 14:43:12.808050 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.807814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.1:43733)
0619 14:43:12.807908 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.807975 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 14:43:12.811671 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.811281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.2:42187)
0619 14:43:12.811522 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.811578 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260619 14:43:12.812402 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.811267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.3:44941)
0619 14:43:12.812276 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.812334 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":937}
I20260619 14:43:12.815338 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.815101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35593 (local address 127.0.98.1:58883)
0619 14:43:12.815199 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.815262 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 14:43:12.819160 7117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:43:12.819317 7117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:43:12.823248 7117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a918948481984a82bef65cbedfad5ef7: Sending DeleteTablet for 3 replicas of tablet 7b5c4bddfb5740cca5de8280c1f3c4d1
I20260619 14:43:12.823956 7690 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b5c4bddfb5740cca5de8280c1f3c4d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:12 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41456
I20260619 14:43:12.824070 7416 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b5c4bddfb5740cca5de8280c1f3c4d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:12 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36640
I20260619 14:43:12.824344 7874 tablet_replica.cc:330] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32: stopping tablet replica
I20260619 14:43:12.824465 7874 raft_consensus.cc:2229] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:43:12.824738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 7264
I20260619 14:43:12.824738 7874 raft_consensus.cc:2258] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:12.825269 7553 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b5c4bddfb5740cca5de8280c1f3c4d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:12 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:34220
I20260619 14:43:12.826000 7874 ts_tablet_manager.cc:1905] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:12.827636 7876 tablet_replica.cc:330] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a: stopping tablet replica
I20260619 14:43:12.827867 7876 raft_consensus.cc:2229] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:43:12.828073 7876 raft_consensus.cc:2258] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:12.828349 7874 ts_tablet_manager.cc:1918] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:43:12.828460 7874 log.cc:1196] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/7b5c4bddfb5740cca5de8280c1f3c4d1
I20260619 14:43:12.828748 7874 ts_tablet_manager.cc:1939] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 875fc9e2d4544467a24fce99d075ed32: Deleting consensus metadata
I20260619 14:43:12.829021 7876 ts_tablet_manager.cc:1905] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:12.829442 7107 catalog_manager.cc:4841] TS 875fc9e2d4544467a24fce99d075ed32 (127.0.98.3:33247): tablet 7b5c4bddfb5740cca5de8280c1f3c4d1 (table test-table [id=d6f7dfca771048c6a71f90b7381e1482]) successfully deleted
I20260619 14:43:12.831068 7876 ts_tablet_manager.cc:1918] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:43:12.831167 7876 log.cc:1196] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/7b5c4bddfb5740cca5de8280c1f3c4d1
I20260619 14:43:12.831414 7876 ts_tablet_manager.cc:1939] T 7b5c4bddfb5740cca5de8280c1f3c4d1 P 23ad98e2a58d4320aaf58238a845e40a: Deleting consensus metadata
I20260619 14:43:12.832073 7106 catalog_manager.cc:4841] TS 23ad98e2a58d4320aaf58238a845e40a (127.0.98.2:34115): tablet 7b5c4bddfb5740cca5de8280c1f3c4d1 (table test-table [id=d6f7dfca771048c6a71f90b7381e1482]) successfully deleted
I20260619 14:43:12.836246 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:12.835993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38627 (local address 127.0.0.1:36646)
0619 14:43:12.836121 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:12.836174 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 14:43:12.836234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 7490
W20260619 14:43:12.837397 7106 catalog_manager.cc:4584] TS bf2f8ba8faff4d7dab0982c9b75bf143 (127.0.98.1:38627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b5c4bddfb5740cca5de8280c1f3c4d1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38627: connect: Connection refused (error 111)
I20260619 14:43:12.842077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 7627
I20260619 14:43:12.848181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 6984
I20260619 14:43:12.853927 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 7090
2026-06-19T14:43:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5415 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:43:12 dist-test-slave-cr22 krb5kdc[7879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:43:12 dist-test-slave-cr22 krb5kdc[7879](info): set up 2 sockets
Jun 19 14:43:12 dist-test-slave-cr22 krb5kdc[7879](info): commencing operation
krb5kdc: starting...
W20260619 14:43:13.908191 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:43:13 dist-test-slave-cr22 krb5kdc[7879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:43:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:43:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 14:43:14.043728 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45811
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:14.160243 7892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:14.160552 7892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:14.160626 7892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:14.164639 7892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:43:14.164739 7892 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:43:14.164773 7892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:14.164813 7892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:14.164850 7892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:43:14.164886 7892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:43:14.170111 7892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45811
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:14.171191 7892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:14.172202 7892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:14.180768 7900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:14.180755 7898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:14.180972 7892 server_base.cc:1034] running on GCE node
W20260619 14:43:14.180755 7897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:14.181674 7892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:14.182870 7892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:14.184087 7892 hybrid_clock.cc:648] HybridClock initialized: now 1781880194184051 us; error 49 us; skew 500 ppm
Jun 19 14:43:14 dist-test-slave-cr22 krb5kdc[7879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:14.187057 7892 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 14:43:14.188089 7892 webserver.cc:460] Webserver started at http://127.0.98.62:37843/ using document root <none> and password file <none>
I20260619 14:43:14.188702 7892 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:14.188797 7892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:14.189029 7892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:14.191116 7892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "c02054cd162d4fbda16a00a5c1525060"
format_stamp: "Formatted at 2026-06-19 14:43:14 on dist-test-slave-cr22"
server_key: "7f770293c344d5abd20eb3fcf0b4aeb5"
server_key_iv: "2d5a62d6c5c84cb94caaaf9318632955"
server_key_version: "encryptionkey@0"
I20260619 14:43:14.191706 7892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "c02054cd162d4fbda16a00a5c1525060"
format_stamp: "Formatted at 2026-06-19 14:43:14 on dist-test-slave-cr22"
server_key: "7f770293c344d5abd20eb3fcf0b4aeb5"
server_key_iv: "2d5a62d6c5c84cb94caaaf9318632955"
server_key_version: "encryptionkey@0"
I20260619 14:43:14.195498 7892 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:43:14.198311 7907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:14.199833 7892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:43:14.199997 7892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "c02054cd162d4fbda16a00a5c1525060"
format_stamp: "Formatted at 2026-06-19 14:43:14 on dist-test-slave-cr22"
server_key: "7f770293c344d5abd20eb3fcf0b4aeb5"
server_key_iv: "2d5a62d6c5c84cb94caaaf9318632955"
server_key_version: "encryptionkey@0"
I20260619 14:43:14.200125 7892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:14.226394 7892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:14.230212 7892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:14.230439 7892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:14.239332 7892 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45811
I20260619 14:43:14.239351 7969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45811 every 8 connection(s)
I20260619 14:43:14.240388 7892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:43:14.240897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 7892
I20260619 14:43:14.241029 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:43:14.241358 392 external_mini_cluster.cc:1428] Setting key 555d28b9e96eff81f82499d6da9e849f
I20260619 14:43:14.243695 7970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:14.249061 7970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060: Bootstrap starting.
Jun 19 14:43:14 dist-test-slave-cr22 krb5kdc[7879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:43:14.252115 7970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:14.253609 7970 log.cc:824] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:14.256106 7970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060: No bootstrap required, opened a new log
I20260619 14:43:14.258060 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:14.242887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48894 (local address 127.0.98.62:45811)
0619 14:43:14.243310 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:14.243325 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:14.243361 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:43:14.244190 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:14.244209 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:14.244549 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:14.244910 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:14.245756 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:14.246820 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:14.247818 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:14.248323 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:14.248502 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:14.251888 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:14.251895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:14.251918 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:14.251969 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:14.254526 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:14.255070 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:14.255079 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:14.255087 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:14.255280 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:14.255692 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:14.255695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:14.255697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:14.256244 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:14.256449 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:14.256768 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 14:43:14.257083 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":123,"threads_started":1}
I20260619 14:43:14.260558 7970 raft_consensus.cc:348] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02054cd162d4fbda16a00a5c1525060" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45811 } }
I20260619 14:43:14.260875 7970 raft_consensus.cc:374] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:14.260943 7970 raft_consensus.cc:729] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c02054cd162d4fbda16a00a5c1525060, State: Initialized, Role: FOLLOWER
I20260619 14:43:14.261493 7970 consensus_queue.cc:260] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02054cd162d4fbda16a00a5c1525060" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45811 } }
I20260619 14:43:14.261642 7970 raft_consensus.cc:388] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:43:14.261689 7970 raft_consensus.cc:482] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:43:14.261802 7970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:14.262881 7970 raft_consensus.cc:504] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02054cd162d4fbda16a00a5c1525060" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45811 } }
I20260619 14:43:14.263269 7970 leader_election.cc:302] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c02054cd162d4fbda16a00a5c1525060; no voters:
I20260619 14:43:14.263669 7970 leader_election.cc:288] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:43:14.263856 7975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:14.264096 7975 raft_consensus.cc:686] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060 [term 1 LEADER]: Becoming Leader. State: Replica: c02054cd162d4fbda16a00a5c1525060, State: Running, Role: LEADER
I20260619 14:43:14.264400 7975 consensus_queue.cc:237] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02054cd162d4fbda16a00a5c1525060" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45811 } }
I20260619 14:43:14.265007 7970 sys_catalog.cc:564] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:43:14.266388 7976 sys_catalog.cc:455] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c02054cd162d4fbda16a00a5c1525060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02054cd162d4fbda16a00a5c1525060" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45811 } } }
I20260619 14:43:14.266533 7976 sys_catalog.cc:458] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:14.266921 7984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:43:14.267671 7977 sys_catalog.cc:455] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c02054cd162d4fbda16a00a5c1525060. Latest consensus state: current_term: 1 leader_uuid: "c02054cd162d4fbda16a00a5c1525060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02054cd162d4fbda16a00a5c1525060" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45811 } } }
I20260619 14:43:14.267825 7977 sys_catalog.cc:458] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:14.270011 7984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:43:14.276319 7984 catalog_manager.cc:1347] Generated new cluster ID: 24ad7132a07240a0a90f3068498e839f
I20260619 14:43:14.276396 7984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:43:14.303334 7984 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:43:14.304687 7984 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:43:14.314697 7984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060: Generated new TSK 0
I20260619 14:43:14.315536 7984 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 14:43:14.348518 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45811
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:14.457697 7996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:14.457969 7996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:14.458045 7996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:14.462051 7996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:14.462138 7996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:14.462208 7996 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:43:14.462289 7996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:43:14.467744 7996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:14.468811 7996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:14.469826 7996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:14.480010 8004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:14.480060 8002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:14.480195 7996 server_base.cc:1034] running on GCE node
W20260619 14:43:14.480005 8001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:14.480677 7996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:14.481353 7996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:14.482569 7996 hybrid_clock.cc:648] HybridClock initialized: now 1781880194482535 us; error 53 us; skew 500 ppm
Jun 19 14:43:14 dist-test-slave-cr22 krb5kdc[7879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:14.485525 7996 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 14:43:14.486366 7996 webserver.cc:460] Webserver started at http://127.0.98.1:33141/ using document root <none> and password file <none>
I20260619 14:43:14.486888 7996 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:14.486946 7996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:14.487107 7996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:14.489358 7996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "0882863e0497441a92182d0f4c12fbe3"
format_stamp: "Formatted at 2026-06-19 14:43:14 on dist-test-slave-cr22"
server_key: "bb071e031ab22d34aa99ebe79257be86"
server_key_iv: "830ae5ab93059e30bd8ecded2bcdc514"
server_key_version: "encryptionkey@0"
I20260619 14:43:14.489880 7996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0882863e0497441a92182d0f4c12fbe3"
format_stamp: "Formatted at 2026-06-19 14:43:14 on dist-test-slave-cr22"
server_key: "bb071e031ab22d34aa99ebe79257be86"
server_key_iv: "830ae5ab93059e30bd8ecded2bcdc514"
server_key_version: "encryptionkey@0"
I20260619 14:43:14.493649 7996 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 14:43:14.496251 8011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:14.497921 7996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:43:14.498031 7996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "0882863e0497441a92182d0f4c12fbe3"
format_stamp: "Formatted at 2026-06-19 14:43:14 on dist-test-slave-cr22"
server_key: "bb071e031ab22d34aa99ebe79257be86"
server_key_iv: "830ae5ab93059e30bd8ecded2bcdc514"
server_key_version: "encryptionkey@0"
I20260619 14:43:14.498131 7996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:14.534678 7996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:14.538039 7996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:14.538218 7996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:14.539795 7996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:14.539848 7996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:14.539891 7996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:14.539906 7996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:43:14 dist-test-slave-cr22 krb5kdc[7879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:43:14.551427 7996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35779
I20260619 14:43:14.551460 8126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35779 every 8 connection(s)
I20260619 14:43:14.552498 7996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:43:14.554713 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:14.540673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54659 (local address 127.0.98.62:45811)
0619 14:43:14.540851 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:14.540859 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:14.541697 (+ 838us) server_negotiation.cc:407] Connection header received
0619 14:43:14.542836 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:14.542842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:14.542909 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:14.543030 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:14.544521 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:14.545440 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:14.546852 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:14.547168 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:14.547292 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:14.550623 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:14.550627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:14.550630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:14.550757 (+ 127us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:14.552363 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:14.552931 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:14.552935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:14.552936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:14.553056 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:14.553481 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:14.553484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:14.553485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:14.553864 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:14.553961 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:14.554476 (+ 515us) server_negotiation.cc:299] Negotiation successful
0619 14:43:14.554589 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:43:14.555334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 7996
I20260619 14:43:14.555513 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:43:14.555689 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:14.541043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45811 (local address 127.0.98.1:54659)
0619 14:43:14.541508 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:14.541558 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:14.542525 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:14.543267 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:14.543275 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:14.543725 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:14.544356 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:14.544372 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:14.545588 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:14.545592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:14.546689 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:14.546699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:14.547356 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:14.547360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:14.547589 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:14.548242 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:14.548266 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:14.550436 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:14.552489 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:14.552495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:14.552509 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:14.552821 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:14.553174 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:14.553177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:14.553179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:14.553344 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:14.553994 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:14.553999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:14.554227 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 14:43:14.554420 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 14:43:14.554676 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":139,"threads_started":1}
I20260619 14:43:14.555884 392 external_mini_cluster.cc:1428] Setting key 912d34293098071e80b3c1cdb87d94ac
I20260619 14:43:14.557063 8127 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45811
I20260619 14:43:14.557220 8127 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:14.557971 8127 heartbeater.cc:511] Master 127.0.98.62:45811 requested a full tablet report, sending...
I20260619 14:43:14.559521 7923 ts_manager.cc:194] Registered new tserver with Master: 0882863e0497441a92182d0f4c12fbe3 (127.0.98.1:35779)
I20260619 14:43:14.560640 7923 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54659
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260619 14:43:14.571507 8134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:14.559252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45811 (local address 127.0.98.1:56189)
0619 14:43:14.563651 (+ 4399us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:14.563664 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:14.563862 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:14.564111 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:14.564114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:14.564312 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:14.564407 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:14.564413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:14.565200 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:14.565203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:14.565795 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:14.565804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:14.566400 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:14.566403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:14.566482 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:14.567022 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:14.567039 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:14.567594 (+ 555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:14.569890 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:14.569893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:14.569894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:14.570091 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:14.570404 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:14.570406 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:14.570408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:14.570482 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:14.570960 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:14.570963 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:14.571049 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 14:43:14.571145 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 14:43:14.571264 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4319,"thread_start_us":128,"threads_started":1}
I20260619 14:43:14.571853 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:14.559330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56189 (local address 127.0.98.62:45811)
0619 14:43:14.559459 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:14.559463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:14.563818 (+ 4355us) server_negotiation.cc:407] Connection header received
0619 14:43:14.563960 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:14.563963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:14.564006 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:14.564094 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:14.564529 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:14.565091 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:14.565919 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:14.566273 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:14.566380 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:14.567750 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:14.567753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:14.567755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:14.567829 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:14.569759 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:14.570192 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:14.570199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:14.570202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:14.570304 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:14.570570 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:14.570573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:14.570575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:14.570833 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:14.570951 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:14.571244 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 14:43:14.571394 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 14:43:14.588040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45811
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:14.701562 8136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:14.701844 8136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:14.701928 8136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:14.706281 8136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:14.706347 8136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:14.706388 8136 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:43:14.706465 8136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:43:14.711746 8136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:14.712869 8136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:14.713965 8136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:14.724319 8141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:14.724355 8142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:14.724362 8144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:14.724520 8136 server_base.cc:1034] running on GCE node
I20260619 14:43:14.725075 8136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:14.725788 8136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:14.726958 8136 hybrid_clock.cc:648] HybridClock initialized: now 1781880194726949 us; error 35 us; skew 500 ppm
Jun 19 14:43:14 dist-test-slave-cr22 krb5kdc[7879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:14.729943 8136 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 14:43:14.731016 8136 webserver.cc:460] Webserver started at http://127.0.98.2:39379/ using document root <none> and password file <none>
I20260619 14:43:14.731673 8136 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:14.731765 8136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:14.732000 8136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:14.734118 8136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "3d49a3b4f9174b3280645f9d1ed45da0"
format_stamp: "Formatted at 2026-06-19 14:43:14 on dist-test-slave-cr22"
server_key: "10c62ca54364f86641e403025520bb9f"
server_key_iv: "a7dac7e6eb53395d87d53a359706694f"
server_key_version: "encryptionkey@0"
I20260619 14:43:14.734712 8136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3d49a3b4f9174b3280645f9d1ed45da0"
format_stamp: "Formatted at 2026-06-19 14:43:14 on dist-test-slave-cr22"
server_key: "10c62ca54364f86641e403025520bb9f"
server_key_iv: "a7dac7e6eb53395d87d53a359706694f"
server_key_version: "encryptionkey@0"
I20260619 14:43:14.738667 8136 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 14:43:14.741477 8151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:14.743278 8136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:43:14.743427 8136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "3d49a3b4f9174b3280645f9d1ed45da0"
format_stamp: "Formatted at 2026-06-19 14:43:14 on dist-test-slave-cr22"
server_key: "10c62ca54364f86641e403025520bb9f"
server_key_iv: "a7dac7e6eb53395d87d53a359706694f"
server_key_version: "encryptionkey@0"
I20260619 14:43:14.743624 8136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:14.759773 8136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:14.763108 8136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:14.763275 8136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:14.765614 8136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:14.765709 8136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:14.765769 8136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:14.765791 8136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:43:14 dist-test-slave-cr22 krb5kdc[7879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:43:14.777621 8136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40441
I20260619 14:43:14.778496 8266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40441 every 8 connection(s)
I20260619 14:43:14.778656 8136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:43:14.780694 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:14.766689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45633 (local address 127.0.98.62:45811)
0619 14:43:14.766906 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:14.766911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:14.767936 (+ 1025us) server_negotiation.cc:407] Connection header received
0619 14:43:14.769086 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:14.769091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:14.769158 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:14.769272 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:14.770669 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:14.771199 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:14.773168 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:14.773449 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:14.773573 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:14.776632 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:14.776636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:14.776638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:14.776671 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:14.778365 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:14.778977 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:14.778980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:14.778981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:14.779107 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:14.779597 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:14.779600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:14.779602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:14.779846 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:14.779932 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:14.780467 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 14:43:14.780574 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260619 14:43:14.781599 8160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:14.767162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45811 (local address 127.0.98.2:45633)
0619 14:43:14.767771 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:14.767822 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:14.768850 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:14.769550 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:14.769559 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:14.769956 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:14.770482 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:14.770499 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:14.771338 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:14.771340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:14.773016 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:14.773029 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:14.773624 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:14.773628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:14.773815 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:14.774479 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:14.774496 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:14.776407 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:14.778499 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:14.778504 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:14.778518 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:14.778873 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:14.779243 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:14.779246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:14.779248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:14.779429 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:14.779956 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:14.779961 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:14.780242 (+ 281us) client_negotiation.cc:769] Sending connection context
0619 14:43:14.780404 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 14:43:14.780689 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":141,"threads_started":1}
I20260619 14:43:14.783174 8267 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45811
I20260619 14:43:14.783404 8267 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:14.784044 8267 heartbeater.cc:511] Master 127.0.98.62:45811 requested a full tablet report, sending...
I20260619 14:43:14.785101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 8136
I20260619 14:43:14.785223 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:43:14.785228 7924 ts_manager.cc:194] Registered new tserver with Master: 3d49a3b4f9174b3280645f9d1ed45da0 (127.0.98.2:40441)
I20260619 14:43:14.785495 392 external_mini_cluster.cc:1428] Setting key 3aec068f694ed24c6bce29287f0a91b5
I20260619 14:43:14.785864 7924 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45633
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:43:14.796028 8273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:14.786155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45811 (local address 127.0.98.2:40871)
0619 14:43:14.787587 (+ 1432us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:14.787600 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:14.787716 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:14.788115 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:14.788118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:14.788327 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:14.788423 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:14.788429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:14.789551 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:14.789554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:14.790186 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:14.790196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:14.790838 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:14.790840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:14.790911 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:14.791528 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:14.791539 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:14.791891 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:14.793921 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:14.793924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:14.793926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:14.794215 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:14.794550 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:14.794553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:14.794554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:14.794647 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:14.795333 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:14.795335 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:14.795419 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 14:43:14.795675 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 14:43:14.795848 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":94,"threads_started":1}
I20260619 14:43:14.796114 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:14.786498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40871 (local address 127.0.98.62:45811)
0619 14:43:14.786652 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:14.786656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:14.787673 (+ 1017us) server_negotiation.cc:407] Connection header received
0619 14:43:14.787968 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:14.787971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:14.788011 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:14.788099 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:14.788534 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:14.789400 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:14.790304 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:14.790713 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:14.790843 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:14.792038 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:14.792044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:14.792050 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:14.792084 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:14.793777 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:14.794326 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:14.794331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:14.794334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:14.794421 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:14.794752 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:14.794755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:14.794757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:14.795075 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:14.795189 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:14.795755 (+ 566us) server_negotiation.cc:299] Negotiation successful
0619 14:43:14.795906 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 14:43:14.818714 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45811
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:14.929641 8276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:14.929881 8276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:14.929924 8276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:14.934016 8276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:14.934070 8276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:14.934109 8276 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:43:14.934177 8276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:43:14.939222 8276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:14.940223 8276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:14.941183 8276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:14.951308 8281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:14.951327 8282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:14.951326 8284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:14.951664 8276 server_base.cc:1034] running on GCE node
I20260619 14:43:14.952051 8276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:14.952664 8276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:14.953857 8276 hybrid_clock.cc:648] HybridClock initialized: now 1781880194953832 us; error 48 us; skew 500 ppm
Jun 19 14:43:14 dist-test-slave-cr22 krb5kdc[7879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:14.956681 8276 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 14:43:14.957516 8276 webserver.cc:460] Webserver started at http://127.0.98.3:43617/ using document root <none> and password file <none>
I20260619 14:43:14.958184 8276 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:14.958276 8276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:14.958483 8276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:14.960505 8276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "a7c724841ed44042a5cbfde5f5177ddd"
format_stamp: "Formatted at 2026-06-19 14:43:14 on dist-test-slave-cr22"
server_key: "5e2982f08d4d9171f0db6d2718c7b8c6"
server_key_iv: "bedaeb4df708899b50f14746bed4c823"
server_key_version: "encryptionkey@0"
I20260619 14:43:14.961035 8276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a7c724841ed44042a5cbfde5f5177ddd"
format_stamp: "Formatted at 2026-06-19 14:43:14 on dist-test-slave-cr22"
server_key: "5e2982f08d4d9171f0db6d2718c7b8c6"
server_key_iv: "bedaeb4df708899b50f14746bed4c823"
server_key_version: "encryptionkey@0"
I20260619 14:43:14.964624 8276 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 14:43:14.967175 8291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:14.968688 8276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:43:14.968794 8276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "a7c724841ed44042a5cbfde5f5177ddd"
format_stamp: "Formatted at 2026-06-19 14:43:14 on dist-test-slave-cr22"
server_key: "5e2982f08d4d9171f0db6d2718c7b8c6"
server_key_iv: "bedaeb4df708899b50f14746bed4c823"
server_key_version: "encryptionkey@0"
I20260619 14:43:14.968884 8276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:15.005800 8276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:15.009496 8276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:15.009677 8276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:15.011194 8276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:15.011282 8276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:15.011365 8276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:15.011417 8276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:43:15 dist-test-slave-cr22 krb5kdc[7879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:43:15.024053 8276 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35841
I20260619 14:43:15.024160 8406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35841 every 8 connection(s)
I20260619 14:43:15.025326 8276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:43:15.025686 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 8276
I20260619 14:43:15.025846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:43:15.026142 392 external_mini_cluster.cc:1428] Setting key 7403a8daa767bb5bdaf1470d32ed92ec
I20260619 14:43:15.027714 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.012160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35885 (local address 127.0.98.62:45811)
0619 14:43:15.012287 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:15.012291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:15.013435 (+ 1144us) server_negotiation.cc:407] Connection header received
0619 14:43:15.014448 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:15.014452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:15.014509 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:15.014573 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:15.016274 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.016932 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.018432 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.018785 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.018916 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.022994 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:15.022998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:15.023002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:15.023033 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:15.025044 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:15.025683 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:15.025691 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:15.025696 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:15.025812 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:15.026405 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:15.026409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:15.026411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:15.026656 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:15.026785 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:15.027347 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 14:43:15.027520 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:43:15.028542 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.012754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45811 (local address 127.0.98.3:35885)
0619 14:43:15.013262 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:15.013304 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:15.014179 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:15.015018 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:15.015025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:15.015556 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:15.016120 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.016134 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.017065 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.017069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.018291 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.018302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.018961 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.018966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.019182 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.019897 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:15.019925 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:15.022738 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:15.025175 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:15.025179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:15.025191 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:15.025546 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:15.026021 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:15.026027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:15.026031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:15.026234 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:15.026795 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:15.026804 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:15.027058 (+ 254us) client_negotiation.cc:769] Sending connection context
0619 14:43:15.027269 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 14:43:15.027659 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":163,"threads_started":1}
I20260619 14:43:15.029630 8407 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45811
I20260619 14:43:15.029763 8407 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:15.030339 8407 heartbeater.cc:511] Master 127.0.98.62:45811 requested a full tablet report, sending...
I20260619 14:43:15.031533 7923 ts_manager.cc:194] Registered new tserver with Master: a7c724841ed44042a5cbfde5f5177ddd (127.0.98.3:35841)
I20260619 14:43:15.032269 7923 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35885
I20260619 14:43:15.039705 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.032081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52649 (local address 127.0.98.62:45811)
0619 14:43:15.032209 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:15.032213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:15.032449 (+ 236us) server_negotiation.cc:407] Connection header received
0619 14:43:15.032589 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:15.032593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:15.032643 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:15.032742 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:15.033238 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.033850 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.034771 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.035023 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.035152 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.036425 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:15.036428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:15.036431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:15.036459 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:15.037884 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:15.038364 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:15.038369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:15.038370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:15.038452 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:15.038736 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:15.038739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:15.038741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:15.038982 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:15.039097 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:15.039399 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 14:43:15.039530 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:43:15.039723 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.031970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45811 (local address 127.0.98.3:52649)
0619 14:43:15.032334 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:15.032376 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:15.032493 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:15.032760 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:15.032763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:15.032982 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:15.033086 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.033092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.033993 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.033996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.034637 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.034646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.035147 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.035151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.035259 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.035915 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:15.035927 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:15.036270 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:15.038033 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:15.038037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:15.038039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:15.038249 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:15.038546 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:15.038550 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:15.038552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:15.038638 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:15.039086 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:15.039089 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:15.039190 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 14:43:15.039343 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 14:43:15.039528 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":102,"threads_started":1}
I20260619 14:43:15.040189 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:43:15 dist-test-slave-cr22 krb5kdc[7879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880195, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:43:15 dist-test-slave-cr22 krb5kdc[7879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:15 dist-test-slave-cr22 krb5kdc[7879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:43:15.066605 7973 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:15.059826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.0.98.62:45811)
0619 14:43:15.059949 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:15.059953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:15.059964 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:43:15.060005 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:15.060007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:15.060045 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:15.060108 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:15.060956 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.061493 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.062473 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.062681 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.062769 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.066444 (+ 3675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48898: BlockingRecv error: recv got EOF from 127.0.0.1:48898 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 14:43:15 dist-test-slave-cr22 krb5kdc[7879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880195, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:43:15.076802 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.068148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48914 (local address 127.0.98.62:45811)
0619 14:43:15.068274 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:15.068277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:15.068322 (+ 45us) server_negotiation.cc:407] Connection header received
0619 14:43:15.068431 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:15.068433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:15.068470 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:15.068539 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:15.069198 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.069678 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.070507 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.070707 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.070790 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.073569 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:15.073572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:15.073574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:15.073596 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:15.075021 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:15.075499 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:15.075502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:15.075503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:15.075580 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:15.075920 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:15.075923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:15.075924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:15.076101 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:15.076219 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:15.076483 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 14:43:15.076623 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:43:15.079929 7923 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:43:15.083190 7923 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:43:15.097757 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.091714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40441 (local address 127.0.0.1:46170)
0619 14:43:15.092332 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:15.092382 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:15.092659 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:15.093263 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:15.093269 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:15.093293 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:15.093445 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.093451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.094668 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.094672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.096301 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.096312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.097187 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.097190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.097292 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.097341 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 14:43:15.097486 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 14:43:15.097610 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":25,"thread_start_us":65,"threads_started":1}
I20260619 14:43:15.097767 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.091848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.0.98.2:40441)
0619 14:43:15.092176 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:15.092187 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:15.092811 (+ 624us) server_negotiation.cc:407] Connection header received
0619 14:43:15.092892 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:15.092897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:15.093113 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:15.093269 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:15.093575 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.094558 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.096442 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.097055 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.097191 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.097361 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:15.097514 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 14:43:15.097610 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":123,"threads_started":1}
I20260619 14:43:15.098349 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.091793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35779 (local address 127.0.0.1:36872)
0619 14:43:15.093037 (+ 1244us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:15.093051 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:15.093136 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:15.093663 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:15.093666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:15.093684 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:15.093778 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.093784 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.095174 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.095178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.096504 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.096512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.097771 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.097776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.097974 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.098033 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 14:43:15.098119 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 14:43:15.098209 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1189,"mutex_wait_us":79,"thread_start_us":90,"threads_started":1}
I20260619 14:43:15.098567 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.092613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43678 (local address 127.0.98.3:35841)
0619 14:43:15.093066 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:15.093078 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:15.093381 (+ 303us) server_negotiation.cc:407] Connection header received
0619 14:43:15.093607 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:15.093614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:15.093781 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:15.093951 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:15.094112 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.095166 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.096494 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.097389 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.097511 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.097701 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:15.098185 (+ 484us) server_negotiation.cc:299] Negotiation successful
0619 14:43:15.098285 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":115,"threads_started":1}
I20260619 14:43:15.098623 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.092515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36872 (local address 127.0.98.1:35779)
0619 14:43:15.092882 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:15.092888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:15.093137 (+ 249us) server_negotiation.cc:407] Connection header received
0619 14:43:15.093332 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:15.093337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:15.093531 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:15.093672 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:15.093903 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.095021 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.096658 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.097485 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.097582 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.097701 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:15.098169 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 14:43:15.098273 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":96,"threads_started":1}
I20260619 14:43:15.099721 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.092457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35841 (local address 127.0.0.1:43678)
0619 14:43:15.093293 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:15.093306 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:15.093439 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:15.093921 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:15.093923 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:15.093939 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:15.094025 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.094030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.095317 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.095321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.096357 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.096367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.097771 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.097776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.097980 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.098036 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 14:43:15.098753 (+ 717us) client_negotiation.cc:240] Negotiation successful
0619 14:43:15.099577 (+ 824us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":83,"thread_start_us":92,"threads_started":1}
I20260619 14:43:15.101133 8189 tablet_service.cc:1467] Processing CreateTablet for tablet 19bff2953ae64099b8b5f3ffe95e79da (DEFAULT_TABLE table=test-table [id=0777c41de0744008a2e0c7e26aefa4f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:15.101428 8061 tablet_service.cc:1467] Processing CreateTablet for tablet 19bff2953ae64099b8b5f3ffe95e79da (DEFAULT_TABLE table=test-table [id=0777c41de0744008a2e0c7e26aefa4f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:15.102281 8189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19bff2953ae64099b8b5f3ffe95e79da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:15.102367 8061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19bff2953ae64099b8b5f3ffe95e79da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:15.102237 8341 tablet_service.cc:1467] Processing CreateTablet for tablet 19bff2953ae64099b8b5f3ffe95e79da (DEFAULT_TABLE table=test-table [id=0777c41de0744008a2e0c7e26aefa4f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:15.103137 8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19bff2953ae64099b8b5f3ffe95e79da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:15.108946 8435 tablet_bootstrap.cc:492] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3: Bootstrap starting.
I20260619 14:43:15.110774 8437 tablet_bootstrap.cc:492] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd: Bootstrap starting.
I20260619 14:43:15.111349 8435 tablet_bootstrap.cc:654] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:15.112149 8435 log.cc:824] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:15.112607 8437 tablet_bootstrap.cc:654] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:15.112852 8436 tablet_bootstrap.cc:492] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0: Bootstrap starting.
I20260619 14:43:15.113479 8437 log.cc:824] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:15.114298 8435 tablet_bootstrap.cc:492] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3: No bootstrap required, opened a new log
I20260619 14:43:15.114518 8435 ts_tablet_manager.cc:1397] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:43:15.115334 8436 tablet_bootstrap.cc:654] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:15.115494 8437 tablet_bootstrap.cc:492] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd: No bootstrap required, opened a new log
I20260619 14:43:15.115676 8437 ts_tablet_manager.cc:1397] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260619 14:43:15.116218 8436 log.cc:824] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:15.118228 8436 tablet_bootstrap.cc:492] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0: No bootstrap required, opened a new log
I20260619 14:43:15.118415 8436 ts_tablet_manager.cc:1397] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:43:15.118374 8435 raft_consensus.cc:348] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.118669 8435 raft_consensus.cc:374] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:15.118722 8435 raft_consensus.cc:729] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0882863e0497441a92182d0f4c12fbe3, State: Initialized, Role: FOLLOWER
I20260619 14:43:15.119189 8435 consensus_queue.cc:260] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.119791 8127 heartbeater.cc:503] Master 127.0.98.62:45811 was elected leader, sending a full tablet report...
I20260619 14:43:15.119737 8437 raft_consensus.cc:348] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.119980 8437 raft_consensus.cc:374] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:15.120029 8437 raft_consensus.cc:729] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7c724841ed44042a5cbfde5f5177ddd, State: Initialized, Role: FOLLOWER
I20260619 14:43:15.120146 8435 ts_tablet_manager.cc:1428] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:43:15.120479 8437 consensus_queue.cc:260] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.121451 8437 ts_tablet_manager.cc:1428] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:43:15.121804 8407 heartbeater.cc:503] Master 127.0.98.62:45811 was elected leader, sending a full tablet report...
I20260619 14:43:15.123202 8436 raft_consensus.cc:348] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.123503 8436 raft_consensus.cc:374] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:15.123564 8436 raft_consensus.cc:729] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d49a3b4f9174b3280645f9d1ed45da0, State: Initialized, Role: FOLLOWER
I20260619 14:43:15.124043 8436 consensus_queue.cc:260] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.124845 8267 heartbeater.cc:503] Master 127.0.98.62:45811 was elected leader, sending a full tablet report...
I20260619 14:43:15.125298 8436 ts_tablet_manager.cc:1428] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260619 14:43:15.276954 8408 tablet.cc:2369] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:43:15.279804 8268 tablet.cc:2369] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:43:15.303974 8128 tablet.cc:2369] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:43:15.309930 8441 raft_consensus.cc:482] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:43:15.310222 8441 raft_consensus.cc:504] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.311601 8441 leader_election.cc:288] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d49a3b4f9174b3280645f9d1ed45da0 (127.0.98.2:40441), a7c724841ed44042a5cbfde5f5177ddd (127.0.98.3:35841)
I20260619 14:43:15.315959 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.312022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54393 (local address 127.0.98.3:35841)
0619 14:43:15.312205 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:15.312214 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:15.312447 (+ 233us) server_negotiation.cc:407] Connection header received
0619 14:43:15.312528 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:15.312530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:15.312706 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:15.312794 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:15.313062 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.313641 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.314836 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.315337 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.315426 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.315559 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:15.315728 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 14:43:15.315822 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 14:43:15.315990 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.311932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35841 (local address 127.0.98.1:54393)
0619 14:43:15.312317 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:15.312328 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:15.312445 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:15.312817 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:15.312819 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:15.312833 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:15.312926 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.312930 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.313794 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.313797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.314701 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.314711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.315487 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.315494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.315586 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.315616 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:43:15.315687 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 14:43:15.315740 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":73,"thread_start_us":113,"threads_started":1}
I20260619 14:43:15.316013 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.311912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42295 (local address 127.0.98.2:40441)
0619 14:43:15.312240 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:15.312246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:15.312265 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:43:15.312426 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:15.312429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:15.312488 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:15.312577 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:15.313013 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.313588 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.314974 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.315358 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.315528 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.315593 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:15.315829 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 14:43:15.315903 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260619 14:43:15.316236 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.311825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40441 (local address 127.0.98.1:42295)
0619 14:43:15.312164 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:15.312181 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:15.312308 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:15.312705 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:15.312710 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:15.312729 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:15.312890 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.312900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.313733 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.313739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.314851 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.314862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.315530 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.315534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.315661 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.315715 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 14:43:15.315800 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 14:43:15.315866 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":104,"thread_start_us":126,"threads_started":1}
I20260619 14:43:15.316705 8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19bff2953ae64099b8b5f3ffe95e79da" candidate_uuid: "0882863e0497441a92182d0f4c12fbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7c724841ed44042a5cbfde5f5177ddd" is_pre_election: true
I20260619 14:43:15.316864 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19bff2953ae64099b8b5f3ffe95e79da" candidate_uuid: "0882863e0497441a92182d0f4c12fbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" is_pre_election: true
I20260619 14:43:15.317049 8361 raft_consensus.cc:2454] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0882863e0497441a92182d0f4c12fbe3 in term 0.
I20260619 14:43:15.317121 8204 raft_consensus.cc:2454] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0882863e0497441a92182d0f4c12fbe3 in term 0.
I20260619 14:43:15.317592 8014 leader_election.cc:302] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0882863e0497441a92182d0f4c12fbe3, 3d49a3b4f9174b3280645f9d1ed45da0; no voters:
I20260619 14:43:15.317910 8441 raft_consensus.cc:2781] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:43:15.317989 8441 raft_consensus.cc:482] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:43:15.318019 8441 raft_consensus.cc:3037] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:15.319149 8441 raft_consensus.cc:504] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.319559 8441 leader_election.cc:288] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [CANDIDATE]: Term 1 election: Requested vote from peers 3d49a3b4f9174b3280645f9d1ed45da0 (127.0.98.2:40441), a7c724841ed44042a5cbfde5f5177ddd (127.0.98.3:35841)
I20260619 14:43:15.319911 8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19bff2953ae64099b8b5f3ffe95e79da" candidate_uuid: "0882863e0497441a92182d0f4c12fbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7c724841ed44042a5cbfde5f5177ddd"
I20260619 14:43:15.319958 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19bff2953ae64099b8b5f3ffe95e79da" candidate_uuid: "0882863e0497441a92182d0f4c12fbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d49a3b4f9174b3280645f9d1ed45da0"
I20260619 14:43:15.320052 8361 raft_consensus.cc:3037] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:15.320086 8204 raft_consensus.cc:3037] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:15.321067 8204 raft_consensus.cc:2454] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0882863e0497441a92182d0f4c12fbe3 in term 1.
I20260619 14:43:15.321411 8014 leader_election.cc:302] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0882863e0497441a92182d0f4c12fbe3, 3d49a3b4f9174b3280645f9d1ed45da0; no voters:
I20260619 14:43:15.321619 8441 raft_consensus.cc:2781] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:15.321725 8361 raft_consensus.cc:2454] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0882863e0497441a92182d0f4c12fbe3 in term 1.
I20260619 14:43:15.321877 8441 raft_consensus.cc:686] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [term 1 LEADER]: Becoming Leader. State: Replica: 0882863e0497441a92182d0f4c12fbe3, State: Running, Role: LEADER
I20260619 14:43:15.322249 8441 consensus_queue.cc:237] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.325309 7923 catalog_manager.cc:5495] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0882863e0497441a92182d0f4c12fbe3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0882863e0497441a92182d0f4c12fbe3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } health_report { overall_health: UNKNOWN } } }
I20260619 14:43:15.389854 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.386441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.0.98.1:35779)
0619 14:43:15.386564 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:15.386575 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:15.386590 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:43:15.386665 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:15.386667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:15.386719 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:15.386805 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:43:15.387022 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.387631 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.388492 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.388692 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.388767 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.389018 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:43:15.389440 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:43:15.389528 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:15.389687 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 14:43:15.389736 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:43:15.395720 8204 raft_consensus.cc:1261] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 FOLLOWER]: Refusing update from remote peer 0882863e0497441a92182d0f4c12fbe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:15.395764 8361 raft_consensus.cc:1261] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd [term 1 FOLLOWER]: Refusing update from remote peer 0882863e0497441a92182d0f4c12fbe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:15.396471 8441 consensus_queue.cc:1035] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:15.396660 8446 consensus_queue.cc:1035] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:15.404408 8449 mvcc.cc:205] Tried to move back new op lower bound from 7298581280333635584 to 7298581280042299392. Current Snapshot: MvccSnapshot[applied={T|T < 7298581280333635584}]
I20260619 14:43:15.405433 8451 mvcc.cc:205] Tried to move back new op lower bound from 7298581280333635584 to 7298581280042299392. Current Snapshot: MvccSnapshot[applied={T|T < 7298581280333635584}]
I20260619 14:43:15.413075 7923 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:43:15.413249 7923 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:43:15.415599 7923 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060: Sending DeleteTablet for 3 replicas of tablet 19bff2953ae64099b8b5f3ffe95e79da
I20260619 14:43:15.416337 8189 tablet_service.cc:1514] Processing DeleteTablet for tablet 19bff2953ae64099b8b5f3ffe95e79da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46170
I20260619 14:43:15.416402 8061 tablet_service.cc:1514] Processing DeleteTablet for tablet 19bff2953ae64099b8b5f3ffe95e79da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36872
I20260619 14:43:15.416558 8341 tablet_service.cc:1514] Processing DeleteTablet for tablet 19bff2953ae64099b8b5f3ffe95e79da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43678
I20260619 14:43:15.416950 8460 tablet_replica.cc:330] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3: stopping tablet replica
I20260619 14:43:15.417002 8461 tablet_replica.cc:330] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd: stopping tablet replica
I20260619 14:43:15.417310 8461 raft_consensus.cc:2229] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:43:15.417312 8460 raft_consensus.cc:2229] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:43:15.417619 8461 raft_consensus.cc:2258] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:15.417657 7923 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:43:15.418074 7923 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:43:15.418459 8460 raft_consensus.cc:2258] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:15.419078 8461 ts_tablet_manager.cc:1905] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:15.419881 8460 ts_tablet_manager.cc:1905] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:15.421851 8461 ts_tablet_manager.cc:1918] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:43:15.421953 8461 log.cc:1196] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/19bff2953ae64099b8b5f3ffe95e79da
I20260619 14:43:15.422266 8461 ts_tablet_manager.cc:1939] T 19bff2953ae64099b8b5f3ffe95e79da P a7c724841ed44042a5cbfde5f5177ddd: Deleting consensus metadata
I20260619 14:43:15.422696 8460 ts_tablet_manager.cc:1918] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:43:15.422860 8460 log.cc:1196] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/wal/wals/19bff2953ae64099b8b5f3ffe95e79da
I20260619 14:43:15.423247 8460 ts_tablet_manager.cc:1939] T 19bff2953ae64099b8b5f3ffe95e79da P 0882863e0497441a92182d0f4c12fbe3: Deleting consensus metadata
I20260619 14:43:15.423280 8061 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3ccbc479244023aa55d0d5e489e7db (DEFAULT_TABLE table=test-table [id=8764482f52ed41378d00fbd7bb4846f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:15.423555 8061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3ccbc479244023aa55d0d5e489e7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:15.423702 8189 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3ccbc479244023aa55d0d5e489e7db (DEFAULT_TABLE table=test-table [id=8764482f52ed41378d00fbd7bb4846f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:15.423976 8189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3ccbc479244023aa55d0d5e489e7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:15.425292 8435 tablet_bootstrap.cc:492] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3: Bootstrap starting.
I20260619 14:43:15.425462 7909 catalog_manager.cc:4841] TS 0882863e0497441a92182d0f4c12fbe3 (127.0.98.1:35779): tablet 19bff2953ae64099b8b5f3ffe95e79da (table test-table [id=0777c41de0744008a2e0c7e26aefa4f8]) successfully deleted
I20260619 14:43:15.426033 7908 catalog_manager.cc:4841] TS a7c724841ed44042a5cbfde5f5177ddd (127.0.98.3:35841): tablet 19bff2953ae64099b8b5f3ffe95e79da (table test-table [id=0777c41de0744008a2e0c7e26aefa4f8]) successfully deleted
I20260619 14:43:15.426239 8435 tablet_bootstrap.cc:654] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:15.426713 8341 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3ccbc479244023aa55d0d5e489e7db (DEFAULT_TABLE table=test-table [id=8764482f52ed41378d00fbd7bb4846f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:15.426837 8436 tablet_bootstrap.cc:492] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0: Bootstrap starting.
I20260619 14:43:15.426975 8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3ccbc479244023aa55d0d5e489e7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:15.427773 8436 tablet_bootstrap.cc:654] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:15.427920 8459 tablet_replica.cc:330] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0: stopping tablet replica
I20260619 14:43:15.428299 8459 raft_consensus.cc:2229] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:43:15.428635 8459 raft_consensus.cc:2258] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:15.429073 8437 tablet_bootstrap.cc:492] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd: Bootstrap starting.
I20260619 14:43:15.429222 8435 tablet_bootstrap.cc:492] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3: No bootstrap required, opened a new log
I20260619 14:43:15.429301 8435 ts_tablet_manager.cc:1397] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 14:43:15.429768 8435 raft_consensus.cc:348] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.429881 8435 raft_consensus.cc:374] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:15.429931 8435 raft_consensus.cc:729] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0882863e0497441a92182d0f4c12fbe3, State: Initialized, Role: FOLLOWER
I20260619 14:43:15.430001 8437 tablet_bootstrap.cc:654] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:15.430070 8435 consensus_queue.cc:260] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.430363 8435 ts_tablet_manager.cc:1428] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:43:15.430974 8436 tablet_bootstrap.cc:492] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0: No bootstrap required, opened a new log
I20260619 14:43:15.431057 8436 ts_tablet_manager.cc:1397] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 14:43:15.431375 8437 tablet_bootstrap.cc:492] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd: No bootstrap required, opened a new log
I20260619 14:43:15.431517 8436 raft_consensus.cc:348] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.431618 8436 raft_consensus.cc:374] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:15.431655 8436 raft_consensus.cc:729] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d49a3b4f9174b3280645f9d1ed45da0, State: Initialized, Role: FOLLOWER
I20260619 14:43:15.431805 8436 consensus_queue.cc:260] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.432062 8436 ts_tablet_manager.cc:1428] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:43:15.432374 8437 ts_tablet_manager.cc:1397] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 14:43:15.432919 8437 raft_consensus.cc:348] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.433034 8437 raft_consensus.cc:374] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:15.433068 8437 raft_consensus.cc:729] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7c724841ed44042a5cbfde5f5177ddd, State: Initialized, Role: FOLLOWER
I20260619 14:43:15.433203 8437 consensus_queue.cc:260] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.434012 8437 ts_tablet_manager.cc:1428] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:43:15.435894 8459 ts_tablet_manager.cc:1905] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:15.438660 8459 ts_tablet_manager.cc:1918] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:43:15.438740 8459 log.cc:1196] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/19bff2953ae64099b8b5f3ffe95e79da
I20260619 14:43:15.439015 8459 ts_tablet_manager.cc:1939] T 19bff2953ae64099b8b5f3ffe95e79da P 3d49a3b4f9174b3280645f9d1ed45da0: Deleting consensus metadata
I20260619 14:43:15.440099 7910 catalog_manager.cc:4841] TS 3d49a3b4f9174b3280645f9d1ed45da0 (127.0.98.2:40441): tablet 19bff2953ae64099b8b5f3ffe95e79da (table test-table [id=0777c41de0744008a2e0c7e26aefa4f8]) successfully deleted
I20260619 14:43:15.478451 8454 raft_consensus.cc:482] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:43:15.478663 8454 raft_consensus.cc:504] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.479264 8454 leader_election.cc:288] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d49a3b4f9174b3280645f9d1ed45da0 (127.0.98.2:40441), a7c724841ed44042a5cbfde5f5177ddd (127.0.98.3:35841)
I20260619 14:43:15.479764 8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3ccbc479244023aa55d0d5e489e7db" candidate_uuid: "0882863e0497441a92182d0f4c12fbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7c724841ed44042a5cbfde5f5177ddd" is_pre_election: true
I20260619 14:43:15.479931 8361 raft_consensus.cc:2454] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0882863e0497441a92182d0f4c12fbe3 in term 0.
I20260619 14:43:15.479909 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3ccbc479244023aa55d0d5e489e7db" candidate_uuid: "0882863e0497441a92182d0f4c12fbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" is_pre_election: true
I20260619 14:43:15.480074 8204 raft_consensus.cc:2454] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0882863e0497441a92182d0f4c12fbe3 in term 0.
I20260619 14:43:15.480480 8012 leader_election.cc:302] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0882863e0497441a92182d0f4c12fbe3, 3d49a3b4f9174b3280645f9d1ed45da0, a7c724841ed44042a5cbfde5f5177ddd; no voters:
I20260619 14:43:15.480741 8454 raft_consensus.cc:2781] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:43:15.480885 8454 raft_consensus.cc:482] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:43:15.480949 8454 raft_consensus.cc:3037] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:15.482093 8454 raft_consensus.cc:504] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.482481 8454 leader_election.cc:288] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [CANDIDATE]: Term 1 election: Requested vote from peers 3d49a3b4f9174b3280645f9d1ed45da0 (127.0.98.2:40441), a7c724841ed44042a5cbfde5f5177ddd (127.0.98.3:35841)
I20260619 14:43:15.482815 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3ccbc479244023aa55d0d5e489e7db" candidate_uuid: "0882863e0497441a92182d0f4c12fbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d49a3b4f9174b3280645f9d1ed45da0"
I20260619 14:43:15.482832 8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3ccbc479244023aa55d0d5e489e7db" candidate_uuid: "0882863e0497441a92182d0f4c12fbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7c724841ed44042a5cbfde5f5177ddd"
I20260619 14:43:15.482949 8204 raft_consensus.cc:3037] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:15.482949 8361 raft_consensus.cc:3037] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:15.484130 8204 raft_consensus.cc:2454] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0882863e0497441a92182d0f4c12fbe3 in term 1.
I20260619 14:43:15.484130 8361 raft_consensus.cc:2454] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0882863e0497441a92182d0f4c12fbe3 in term 1.
I20260619 14:43:15.484426 8012 leader_election.cc:302] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0882863e0497441a92182d0f4c12fbe3, a7c724841ed44042a5cbfde5f5177ddd; no voters:
I20260619 14:43:15.484624 8454 raft_consensus.cc:2781] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:15.484726 8454 raft_consensus.cc:686] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [term 1 LEADER]: Becoming Leader. State: Replica: 0882863e0497441a92182d0f4c12fbe3, State: Running, Role: LEADER
I20260619 14:43:15.484875 8454 consensus_queue.cc:237] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.486646 7924 catalog_manager.cc:5495] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0882863e0497441a92182d0f4c12fbe3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0882863e0497441a92182d0f4c12fbe3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:43:15 dist-test-slave-cr22 krb5kdc[7879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:43:15.530871 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.521265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45811 (local address 127.0.0.1:48916)
0619 14:43:15.521704 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:15.521716 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:15.521819 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:15.522086 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:15.522088 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:15.522458 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:15.522551 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.522557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.523660 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.523663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.524214 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.524222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.524772 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.524775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.524836 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.525443 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:15.525455 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:15.527027 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:15.529039 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:15.529051 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:15.529057 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:15.529333 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:15.529678 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:15.529680 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:15.529682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:15.529761 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:15.530244 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:15.530249 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:15.530349 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 14:43:15.530498 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 14:43:15.530718 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":4,"thread_start_us":107,"threads_started":1}
I20260619 14:43:15.531038 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.521448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48916 (local address 127.0.98.62:45811)
0619 14:43:15.521582 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:15.521590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:15.521789 (+ 199us) server_negotiation.cc:407] Connection header received
0619 14:43:15.521897 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:15.521900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:15.521951 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:15.522048 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:15.522668 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.523516 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.524342 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.524651 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.524763 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.527164 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:15.527168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:15.527171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:15.527197 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:15.528918 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:15.529438 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:15.529442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:15.529444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:15.529568 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:15.529856 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:15.529859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:15.529861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:15.530118 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:15.530231 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:15.530540 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 14:43:15.530848 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":124,"server-negotiator.queue_time_us":53}
I20260619 14:43:15.535742 7924 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48916:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:43:15.536543 7924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:43:15.542055 8061 tablet_service.cc:1467] Processing CreateTablet for tablet 93f4e67e598d4b0386683eeea44a8aab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f97cea5d47f5438ab1b6fb5134cedc1f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:43:15.542390 8189 tablet_service.cc:1467] Processing CreateTablet for tablet 93f4e67e598d4b0386683eeea44a8aab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f97cea5d47f5438ab1b6fb5134cedc1f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:43:15.542515 8061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93f4e67e598d4b0386683eeea44a8aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:15.542770 8189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93f4e67e598d4b0386683eeea44a8aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:15.544138 8341 tablet_service.cc:1467] Processing CreateTablet for tablet 93f4e67e598d4b0386683eeea44a8aab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f97cea5d47f5438ab1b6fb5134cedc1f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:43:15.544525 8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93f4e67e598d4b0386683eeea44a8aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:15.545751 8435 tablet_bootstrap.cc:492] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3: Bootstrap starting.
I20260619 14:43:15.545874 8436 tablet_bootstrap.cc:492] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0: Bootstrap starting.
I20260619 14:43:15.546783 8435 tablet_bootstrap.cc:654] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:15.546968 8436 tablet_bootstrap.cc:654] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:15.548327 8435 tablet_bootstrap.cc:492] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3: No bootstrap required, opened a new log
I20260619 14:43:15.548399 8436 tablet_bootstrap.cc:492] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0: No bootstrap required, opened a new log
I20260619 14:43:15.548424 8435 ts_tablet_manager.cc:1397] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:43:15.548481 8436 ts_tablet_manager.cc:1397] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:43:15.548985 8436 raft_consensus.cc:348] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.549124 8436 raft_consensus.cc:374] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:15.549173 8436 raft_consensus.cc:729] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d49a3b4f9174b3280645f9d1ed45da0, State: Initialized, Role: FOLLOWER
I20260619 14:43:15.549175 8435 raft_consensus.cc:348] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.549286 8435 raft_consensus.cc:374] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:15.549320 8435 raft_consensus.cc:729] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0882863e0497441a92182d0f4c12fbe3, State: Initialized, Role: FOLLOWER
I20260619 14:43:15.549304 8436 consensus_queue.cc:260] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.549444 8435 consensus_queue.cc:260] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.549602 8436 ts_tablet_manager.cc:1428] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:43:15.549546 8443 tablet_replica.cc:439] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } } }
I20260619 14:43:15.549700 8443 tablet_replica.cc:442] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:43:15.549736 8435 ts_tablet_manager.cc:1428] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 14:43:15.549983 8454 tablet_replica.cc:439] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } } }
I20260619 14:43:15.550107 8454 tablet_replica.cc:442] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:43:15.555049 8437 tablet_bootstrap.cc:492] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd: Bootstrap starting.
I20260619 14:43:15.556036 8437 tablet_bootstrap.cc:654] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:15.557489 8437 tablet_bootstrap.cc:492] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd: No bootstrap required, opened a new log
I20260619 14:43:15.557579 8437 ts_tablet_manager.cc:1397] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:43:15.558046 8437 raft_consensus.cc:348] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.558184 8437 raft_consensus.cc:374] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:15.558228 8437 raft_consensus.cc:729] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7c724841ed44042a5cbfde5f5177ddd, State: Initialized, Role: FOLLOWER
I20260619 14:43:15.558425 8437 consensus_queue.cc:260] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.558679 8442 tablet_replica.cc:439] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } } }
I20260619 14:43:15.558845 8442 tablet_replica.cc:442] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:43:15.559746 8437 ts_tablet_manager.cc:1428] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:43:15.882823 8442 raft_consensus.cc:482] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:43:15.882968 8442 raft_consensus.cc:504] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.884254 8442 leader_election.cc:288] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0882863e0497441a92182d0f4c12fbe3 (127.0.98.1:35779), 3d49a3b4f9174b3280645f9d1ed45da0 (127.0.98.2:40441)
I20260619 14:43:15.885841 8443 raft_consensus.cc:482] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:43:15.885962 8443 raft_consensus.cc:504] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.887086 8443 leader_election.cc:288] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0882863e0497441a92182d0f4c12fbe3 (127.0.98.1:35779), a7c724841ed44042a5cbfde5f5177ddd (127.0.98.3:35841)
I20260619 14:43:15.889487 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.884797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33149 (local address 127.0.98.1:35779)
0619 14:43:15.884937 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:15.884943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:15.885043 (+ 100us) server_negotiation.cc:407] Connection header received
0619 14:43:15.885205 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:15.885209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:15.885271 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:15.885348 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:15.885718 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.886538 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.888234 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.888869 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.888979 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.889049 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:15.889278 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 14:43:15.889354 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":36}
I20260619 14:43:15.889670 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.884571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40441 (local address 127.0.98.3:56147)
0619 14:43:15.885046 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:15.885063 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:15.885209 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:15.885511 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:15.885513 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:15.885528 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:15.885665 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.885675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.886687 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.886689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.888243 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.888250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.889199 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.889204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.889344 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.889369 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:43:15.889419 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 14:43:15.889492 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":59,"thread_start_us":46,"threads_started":1}
I20260619 14:43:15.889922 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.884558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35779 (local address 127.0.98.3:33149)
0619 14:43:15.884935 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:15.884951 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:15.885070 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:15.885383 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:15.885387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:15.885403 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:15.885542 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.885552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.886679 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.886683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.888083 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.888102 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.888989 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.888994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.889128 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.889154 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:43:15.889674 (+ 520us) client_negotiation.cc:240] Negotiation successful
0619 14:43:15.889757 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":145,"thread_start_us":89,"threads_started":1}
I20260619 14:43:15.890177 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.884697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56147 (local address 127.0.98.2:40441)
0619 14:43:15.884937 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:15.884943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:15.885196 (+ 253us) server_negotiation.cc:407] Connection header received
0619 14:43:15.885336 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:15.885342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:15.885398 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:15.885482 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:15.885794 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.886505 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.888335 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.889060 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.889863 (+ 803us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.889930 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:15.889965 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 14:43:15.890032 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
I20260619 14:43:15.890425 8081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93f4e67e598d4b0386683eeea44a8aab" candidate_uuid: "a7c724841ed44042a5cbfde5f5177ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0882863e0497441a92182d0f4c12fbe3" is_pre_election: true
I20260619 14:43:15.890614 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93f4e67e598d4b0386683eeea44a8aab" candidate_uuid: "a7c724841ed44042a5cbfde5f5177ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" is_pre_election: true
I20260619 14:43:15.890650 8081 raft_consensus.cc:2454] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7c724841ed44042a5cbfde5f5177ddd in term 0.
I20260619 14:43:15.890745 8204 raft_consensus.cc:2454] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7c724841ed44042a5cbfde5f5177ddd in term 0.
I20260619 14:43:15.891068 8294 leader_election.cc:302] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0882863e0497441a92182d0f4c12fbe3, 3d49a3b4f9174b3280645f9d1ed45da0, a7c724841ed44042a5cbfde5f5177ddd; no voters:
I20260619 14:43:15.891332 8442 raft_consensus.cc:2781] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:43:15.891386 8442 raft_consensus.cc:482] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:43:15.891423 8442 raft_consensus.cc:3037] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:15.892628 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.887260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35779 (local address 127.0.98.2:56015)
0619 14:43:15.887585 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:15.887602 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:15.887696 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:15.888261 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:15.888268 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:15.888285 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:15.888415 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.888421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.889810 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.889817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.891125 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.891134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.892129 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.892132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.892223 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.892248 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:43:15.892329 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:43:15.892396 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":22,"thread_start_us":75,"threads_started":1}
I20260619 14:43:15.892648 8442 raft_consensus.cc:504] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.893028 8442 leader_election.cc:288] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [CANDIDATE]: Term 1 election: Requested vote from peers 0882863e0497441a92182d0f4c12fbe3 (127.0.98.1:35779), 3d49a3b4f9174b3280645f9d1ed45da0 (127.0.98.2:40441)
I20260619 14:43:15.893128 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.887366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56015 (local address 127.0.98.1:35779)
0619 14:43:15.887994 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:15.887998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:15.888013 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:43:15.888058 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:15.888062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:15.888131 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:15.888515 (+ 384us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:15.888612 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.889665 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.891262 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.891992 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.892112 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.892169 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:15.892904 (+ 735us) server_negotiation.cc:299] Negotiation successful
0619 14:43:15.892989 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":553,"thread_start_us":81,"threads_started":1}
I20260619 14:43:15.893404 8080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93f4e67e598d4b0386683eeea44a8aab" candidate_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0882863e0497441a92182d0f4c12fbe3" is_pre_election: true
I20260619 14:43:15.893534 8080 raft_consensus.cc:2454] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d49a3b4f9174b3280645f9d1ed45da0 in term 0.
I20260619 14:43:15.893563 8081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93f4e67e598d4b0386683eeea44a8aab" candidate_uuid: "a7c724841ed44042a5cbfde5f5177ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0882863e0497441a92182d0f4c12fbe3"
I20260619 14:43:15.893636 8081 raft_consensus.cc:3037] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:15.893605 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93f4e67e598d4b0386683eeea44a8aab" candidate_uuid: "a7c724841ed44042a5cbfde5f5177ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d49a3b4f9174b3280645f9d1ed45da0"
I20260619 14:43:15.893739 8204 raft_consensus.cc:3037] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:15.893934 8153 leader_election.cc:302] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0882863e0497441a92182d0f4c12fbe3, 3d49a3b4f9174b3280645f9d1ed45da0; no voters:
I20260619 14:43:15.894361 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.887700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35841 (local address 127.0.98.2:37989)
0619 14:43:15.888047 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:15.888060 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:15.888144 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:15.888669 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:15.888672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:15.888689 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:15.888779 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.888785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.890884 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.890890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.892635 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.892644 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.893898 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.893905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.894028 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.894053 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:43:15.894146 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 14:43:15.894216 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":91,"threads_started":1}
I20260619 14:43:15.894743 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.887909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37989 (local address 127.0.98.3:35841)
0619 14:43:15.888418 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:15.888423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:15.888433 (+ 10us) server_negotiation.cc:407] Connection header received
0619 14:43:15.888464 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:15.888466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:15.888501 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:15.888569 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:15.888897 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.890631 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.892830 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.893626 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.893741 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.893813 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:15.894510 (+ 697us) server_negotiation.cc:299] Negotiation successful
0619 14:43:15.894589 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":452,"thread_start_us":50,"threads_started":1}
I20260619 14:43:15.894869 8081 raft_consensus.cc:2454] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7c724841ed44042a5cbfde5f5177ddd in term 1.
I20260619 14:43:15.894987 8204 raft_consensus.cc:2454] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7c724841ed44042a5cbfde5f5177ddd in term 1.
I20260619 14:43:15.895102 8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93f4e67e598d4b0386683eeea44a8aab" candidate_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7c724841ed44042a5cbfde5f5177ddd" is_pre_election: true
I20260619 14:43:15.895171 8293 leader_election.cc:302] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0882863e0497441a92182d0f4c12fbe3, a7c724841ed44042a5cbfde5f5177ddd; no voters:
I20260619 14:43:15.895262 8361 raft_consensus.cc:2379] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d49a3b4f9174b3280645f9d1ed45da0 in current term 1: Already voted for candidate a7c724841ed44042a5cbfde5f5177ddd in this term.
I20260619 14:43:15.895272 8443 raft_consensus.cc:2741] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 14:43:15.895332 8442 raft_consensus.cc:2781] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:15.895562 8442 raft_consensus.cc:686] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [term 1 LEADER]: Becoming Leader. State: Replica: a7c724841ed44042a5cbfde5f5177ddd, State: Running, Role: LEADER
I20260619 14:43:15.895808 8442 consensus_queue.cc:237] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:15.897287 8482 tablet_replica.cc:439] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd: TxnStatusTablet state changed. Reason: New leader a7c724841ed44042a5cbfde5f5177ddd. Latest consensus state: current_term: 1 leader_uuid: "a7c724841ed44042a5cbfde5f5177ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } } }
I20260619 14:43:15.897385 8482 tablet_replica.cc:442] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:43:15.898046 8484 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:43:15.898172 8484 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:43:15.898474 7924 catalog_manager.cc:5495] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a7c724841ed44042a5cbfde5f5177ddd (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a7c724841ed44042a5cbfde5f5177ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } health_report { overall_health: HEALTHY } } }
I20260619 14:43:15.920265 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.916098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35841 (local address 127.0.0.1:43692)
0619 14:43:15.916259 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:15.916276 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:15.916375 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:15.916697 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:15.916699 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:15.916705 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:43:15.916794 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.916798 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.917797 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.917800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.918441 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.918450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.919021 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.919024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.919106 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.919149 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:43:15.919975 (+ 826us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:43:15.919989 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 14:43:15.920051 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 14:43:15.920126 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 14:43:15.920261 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.916227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43692 (local address 127.0.98.3:35841)
0619 14:43:15.916359 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:15.916363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:15.916380 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:43:15.916503 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:15.916506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:15.916577 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:15.916679 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:43:15.916896 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.917672 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.918559 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.918910 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.919013 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.919319 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:43:15.919873 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:43:15.919987 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:15.920064 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 14:43:15.920119 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:43:15.923329 8204 raft_consensus.cc:1261] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 FOLLOWER]: Refusing update from remote peer a7c724841ed44042a5cbfde5f5177ddd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:15.923569 8081 raft_consensus.cc:1261] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3 [term 1 FOLLOWER]: Refusing update from remote peer a7c724841ed44042a5cbfde5f5177ddd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:15.923935 8482 consensus_queue.cc:1035] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:15.924118 8442 consensus_queue.cc:1035] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:15.925547 8443 tablet_replica.cc:439] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0: TxnStatusTablet state changed. Reason: New leader a7c724841ed44042a5cbfde5f5177ddd. Latest consensus state: current_term: 1 leader_uuid: "a7c724841ed44042a5cbfde5f5177ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } } }
I20260619 14:43:15.925673 8443 tablet_replica.cc:442] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:43:15.925719 8454 tablet_replica.cc:439] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3: TxnStatusTablet state changed. Reason: New leader a7c724841ed44042a5cbfde5f5177ddd. Latest consensus state: current_term: 1 leader_uuid: "a7c724841ed44042a5cbfde5f5177ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } } }
I20260619 14:43:15.925837 8454 tablet_replica.cc:442] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:43:15.928066 8482 tablet_replica.cc:439] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7c724841ed44042a5cbfde5f5177ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } } }
I20260619 14:43:15.928196 8482 tablet_replica.cc:442] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:43:15.928732 8482 tablet_replica.cc:439] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7c724841ed44042a5cbfde5f5177ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } } }
I20260619 14:43:15.928828 8482 tablet_replica.cc:442] T 93f4e67e598d4b0386683eeea44a8aab P a7c724841ed44042a5cbfde5f5177ddd: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:43:15.929723 8443 tablet_replica.cc:439] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7c724841ed44042a5cbfde5f5177ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } } }
I20260619 14:43:15.929826 8443 tablet_replica.cc:442] T 93f4e67e598d4b0386683eeea44a8aab P 3d49a3b4f9174b3280645f9d1ed45da0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:43:15.932789 8454 tablet_replica.cc:439] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7c724841ed44042a5cbfde5f5177ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } } }
I20260619 14:43:15.932917 8454 tablet_replica.cc:442] T 93f4e67e598d4b0386683eeea44a8aab P 0882863e0497441a92182d0f4c12fbe3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:43:15.944319 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.940429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35841 (local address 127.0.98.1:50113)
0619 14:43:15.940804 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:15.940818 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:15.940954 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:15.941274 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:15.941277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:15.941285 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:43:15.941405 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.941411 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.942246 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.942250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.942979 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.942988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.943490 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.943493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.943606 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.943655 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:43:15.943998 (+ 343us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:43:15.944005 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:43:15.944072 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 14:43:15.944178 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":101,"threads_started":1}
I20260619 14:43:15.944300 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.940580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50113 (local address 127.0.98.3:35841)
0619 14:43:15.940722 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:15.940727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:15.940897 (+ 170us) server_negotiation.cc:407] Connection header received
0619 14:43:15.941077 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:15.941080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:15.941161 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:15.941250 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:43:15.941518 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.942106 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.943120 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.943358 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.943500 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.943806 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:43:15.943893 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:43:15.943978 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:15.944090 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 14:43:15.944145 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:43:15.951305 8204 raft_consensus.cc:1261] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 FOLLOWER]: Refusing update from remote peer 0882863e0497441a92182d0f4c12fbe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:15.951326 8361 raft_consensus.cc:1261] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd [term 1 FOLLOWER]: Refusing update from remote peer 0882863e0497441a92182d0f4c12fbe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:15.951728 8454 consensus_queue.cc:1035] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:15.951893 8441 consensus_queue.cc:1035] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:15.957134 8488 mvcc.cc:205] Tried to move back new op lower bound from 7298581282612916224 to 7298581280707256320. Current Snapshot: MvccSnapshot[applied={T|T < 7298581282612916224}]
I20260619 14:43:15.973218 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.969418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46049 (local address 127.0.98.1:35779)
0619 14:43:15.969525 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:15.969531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:15.969774 (+ 243us) server_negotiation.cc:407] Connection header received
0619 14:43:15.969941 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:15.969944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:15.969989 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:15.970074 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:43:15.970357 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.970868 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.972024 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.972330 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.972462 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.972800 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:43:15.972886 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:43:15.972946 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:15.973058 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 14:43:15.973102 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 14:43:15.973328 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:15.969364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35779 (local address 127.0.98.3:46049)
0619 14:43:15.969680 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:15.969700 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:15.969829 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:15.970092 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:15.970095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:15.970103 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:43:15.970253 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.970259 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.970977 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.970981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.971881 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:15.971892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:15.972472 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:15.972475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:15.972578 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:15.972631 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:43:15.972971 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:43:15.972977 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:43:15.973051 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:43:15.973142 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":110,"threads_started":1}
I20260619 14:43:15.997434 7923 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:43:15.997687 7923 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:43:15.999943 7923 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060: Sending DeleteTablet for 3 replicas of tablet 1b3ccbc479244023aa55d0d5e489e7db
I20260619 14:43:16.000504 8061 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b3ccbc479244023aa55d0d5e489e7db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36872
I20260619 14:43:16.000962 8189 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b3ccbc479244023aa55d0d5e489e7db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46170
I20260619 14:43:16.001006 8505 tablet_replica.cc:330] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3: stopping tablet replica
I20260619 14:43:16.001154 8505 raft_consensus.cc:2229] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:43:16.001304 8506 tablet_replica.cc:330] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0: stopping tablet replica
I20260619 14:43:16.001437 8506 raft_consensus.cc:2229] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:43:16.001430 8505 raft_consensus.cc:2258] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:16.001434 8341 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b3ccbc479244023aa55d0d5e489e7db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43678
I20260619 14:43:16.001545 8506 raft_consensus.cc:2258] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:16.002223 8505 ts_tablet_manager.cc:1905] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:16.002347 8506 ts_tablet_manager.cc:1905] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:16.005935 8506 ts_tablet_manager.cc:1918] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:43:16.005944 8505 ts_tablet_manager.cc:1918] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:43:16.006019 8505 log.cc:1196] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/wal/wals/1b3ccbc479244023aa55d0d5e489e7db
I20260619 14:43:16.006022 8506 log.cc:1196] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/1b3ccbc479244023aa55d0d5e489e7db
I20260619 14:43:16.006287 8506 ts_tablet_manager.cc:1939] T 1b3ccbc479244023aa55d0d5e489e7db P 3d49a3b4f9174b3280645f9d1ed45da0: Deleting consensus metadata
I20260619 14:43:16.006330 8505 ts_tablet_manager.cc:1939] T 1b3ccbc479244023aa55d0d5e489e7db P 0882863e0497441a92182d0f4c12fbe3: Deleting consensus metadata
I20260619 14:43:16.007210 7910 catalog_manager.cc:4841] TS 3d49a3b4f9174b3280645f9d1ed45da0 (127.0.98.2:40441): tablet 1b3ccbc479244023aa55d0d5e489e7db (table test-table [id=8764482f52ed41378d00fbd7bb4846f4]) successfully deleted
I20260619 14:43:16.007412 7909 catalog_manager.cc:4841] TS 0882863e0497441a92182d0f4c12fbe3 (127.0.98.1:35779): tablet 1b3ccbc479244023aa55d0d5e489e7db (table test-table [id=8764482f52ed41378d00fbd7bb4846f4]) successfully deleted
I20260619 14:43:16.010442 8507 tablet_replica.cc:330] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd: stopping tablet replica
I20260619 14:43:16.010670 8507 raft_consensus.cc:2229] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:43:16.010851 8507 raft_consensus.cc:2258] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:16.011790 8507 ts_tablet_manager.cc:1905] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:16.015113 8507 ts_tablet_manager.cc:1918] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:43:16.015210 8507 log.cc:1196] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/1b3ccbc479244023aa55d0d5e489e7db
I20260619 14:43:16.015544 8507 ts_tablet_manager.cc:1939] T 1b3ccbc479244023aa55d0d5e489e7db P a7c724841ed44042a5cbfde5f5177ddd: Deleting consensus metadata
I20260619 14:43:16.016467 7908 catalog_manager.cc:4841] TS a7c724841ed44042a5cbfde5f5177ddd (127.0.98.3:35841): tablet 1b3ccbc479244023aa55d0d5e489e7db (table test-table [id=8764482f52ed41378d00fbd7bb4846f4]) successfully deleted
Jun 19 14:43:16 dist-test-slave-cr22 krb5kdc[7879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:43:16 dist-test-slave-cr22 krb5kdc[7879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:16 dist-test-slave-cr22 krb5kdc[7879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:43:16.034446 7973 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:16.027069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48918 (local address 127.0.98.62:45811)
0619 14:43:16.027244 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:16.027249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:16.027401 (+ 152us) server_negotiation.cc:407] Connection header received
0619 14:43:16.027620 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:16.027623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:16.027684 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:16.027790 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:16.028647 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:16.029364 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:16.030450 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:16.030702 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:16.030837 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:16.034328 (+ 3491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48918: BlockingRecv error: recv got EOF from 127.0.0.1:48918 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 14:43:16 dist-test-slave-cr22 krb5kdc[7879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:43:16.049589 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:16.040029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.0.98.62:45811)
0619 14:43:16.040159 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:16.040164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:16.040376 (+ 212us) server_negotiation.cc:407] Connection header received
0619 14:43:16.040586 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:16.040589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:16.040644 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:16.040739 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:16.041415 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:16.041990 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:16.042869 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:16.043111 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:16.043230 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:16.045882 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:16.045886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:16.045889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:16.045917 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:16.047657 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:16.048094 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:16.048100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:16.048106 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:16.048206 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:16.048545 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:16.048548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:16.048550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:16.048741 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:16.048907 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:16.049159 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 14:43:16.049288 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:43:16.052587 7923 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:43:16.053063 7923 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:43:16.058481 8061 tablet_service.cc:1467] Processing CreateTablet for tablet 232cf26c62534739ab0b7e22a32064ae (DEFAULT_TABLE table=test-table [id=cc5c07eb571648838c152f0f27ae93b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:16.058777 8061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 232cf26c62534739ab0b7e22a32064ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:16.059731 8189 tablet_service.cc:1467] Processing CreateTablet for tablet 232cf26c62534739ab0b7e22a32064ae (DEFAULT_TABLE table=test-table [id=cc5c07eb571648838c152f0f27ae93b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:16.060007 8189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 232cf26c62534739ab0b7e22a32064ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:16.061106 8520 tablet_bootstrap.cc:492] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3: Bootstrap starting.
I20260619 14:43:16.061928 8520 tablet_bootstrap.cc:654] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:16.062333 8521 tablet_bootstrap.cc:492] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0: Bootstrap starting.
I20260619 14:43:16.063247 8341 tablet_service.cc:1467] Processing CreateTablet for tablet 232cf26c62534739ab0b7e22a32064ae (DEFAULT_TABLE table=test-table [id=cc5c07eb571648838c152f0f27ae93b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:16.063602 8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 232cf26c62534739ab0b7e22a32064ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:16.065977 8523 tablet_bootstrap.cc:492] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd: Bootstrap starting.
I20260619 14:43:16.066370 8521 tablet_bootstrap.cc:654] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:16.067255 8523 tablet_bootstrap.cc:654] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:16.068060 8520 tablet_bootstrap.cc:492] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3: No bootstrap required, opened a new log
I20260619 14:43:16.068166 8520 ts_tablet_manager.cc:1397] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260619 14:43:16.068675 8520 raft_consensus.cc:348] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:16.068809 8520 raft_consensus.cc:374] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:16.068863 8520 raft_consensus.cc:729] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0882863e0497441a92182d0f4c12fbe3, State: Initialized, Role: FOLLOWER
I20260619 14:43:16.068938 8523 tablet_bootstrap.cc:492] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd: No bootstrap required, opened a new log
I20260619 14:43:16.069016 8523 ts_tablet_manager.cc:1397] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:43:16.069380 8521 tablet_bootstrap.cc:492] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0: No bootstrap required, opened a new log
I20260619 14:43:16.069483 8521 ts_tablet_manager.cc:1397] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260619 14:43:16.069465 8523 raft_consensus.cc:348] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:16.069578 8523 raft_consensus.cc:374] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:16.069625 8523 raft_consensus.cc:729] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7c724841ed44042a5cbfde5f5177ddd, State: Initialized, Role: FOLLOWER
I20260619 14:43:16.069753 8523 consensus_queue.cc:260] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:16.069975 8521 raft_consensus.cc:348] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:16.070092 8521 raft_consensus.cc:374] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:16.070132 8521 raft_consensus.cc:729] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d49a3b4f9174b3280645f9d1ed45da0, State: Initialized, Role: FOLLOWER
I20260619 14:43:16.070134 8523 ts_tablet_manager.cc:1428] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:43:16.070268 8521 consensus_queue.cc:260] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:16.071302 8521 ts_tablet_manager.cc:1428] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:43:16.071535 8520 consensus_queue.cc:260] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:16.072013 8520 ts_tablet_manager.cc:1428] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 14:43:16.133613 8443 raft_consensus.cc:482] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:43:16.133802 8443 raft_consensus.cc:504] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:16.134306 8443 leader_election.cc:288] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0882863e0497441a92182d0f4c12fbe3 (127.0.98.1:35779), a7c724841ed44042a5cbfde5f5177ddd (127.0.98.3:35841)
I20260619 14:43:16.134739 8081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232cf26c62534739ab0b7e22a32064ae" candidate_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0882863e0497441a92182d0f4c12fbe3" is_pre_election: true
I20260619 14:43:16.134882 8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232cf26c62534739ab0b7e22a32064ae" candidate_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7c724841ed44042a5cbfde5f5177ddd" is_pre_election: true
I20260619 14:43:16.134985 8081 raft_consensus.cc:2454] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d49a3b4f9174b3280645f9d1ed45da0 in term 0.
I20260619 14:43:16.135087 8361 raft_consensus.cc:2454] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d49a3b4f9174b3280645f9d1ed45da0 in term 0.
I20260619 14:43:16.135368 8153 leader_election.cc:302] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0882863e0497441a92182d0f4c12fbe3, 3d49a3b4f9174b3280645f9d1ed45da0; no voters:
I20260619 14:43:16.135576 8443 raft_consensus.cc:2781] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:43:16.135641 8443 raft_consensus.cc:482] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:43:16.135690 8443 raft_consensus.cc:3037] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:16.136756 8443 raft_consensus.cc:504] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:16.137104 8443 leader_election.cc:288] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [CANDIDATE]: Term 1 election: Requested vote from peers 0882863e0497441a92182d0f4c12fbe3 (127.0.98.1:35779), a7c724841ed44042a5cbfde5f5177ddd (127.0.98.3:35841)
I20260619 14:43:16.137424 8081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232cf26c62534739ab0b7e22a32064ae" candidate_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0882863e0497441a92182d0f4c12fbe3"
I20260619 14:43:16.137498 8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232cf26c62534739ab0b7e22a32064ae" candidate_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7c724841ed44042a5cbfde5f5177ddd"
I20260619 14:43:16.137583 8081 raft_consensus.cc:3037] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:16.137620 8361 raft_consensus.cc:3037] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:16.138473 8361 raft_consensus.cc:2454] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d49a3b4f9174b3280645f9d1ed45da0 in term 1.
I20260619 14:43:16.138640 8081 raft_consensus.cc:2454] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d49a3b4f9174b3280645f9d1ed45da0 in term 1.
I20260619 14:43:16.138860 8152 leader_election.cc:302] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d49a3b4f9174b3280645f9d1ed45da0, a7c724841ed44042a5cbfde5f5177ddd; no voters:
I20260619 14:43:16.139060 8443 raft_consensus.cc:2781] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:16.139267 8443 raft_consensus.cc:686] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 LEADER]: Becoming Leader. State: Replica: 3d49a3b4f9174b3280645f9d1ed45da0, State: Running, Role: LEADER
I20260619 14:43:16.139550 8443 consensus_queue.cc:237] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:16.141904 7924 catalog_manager.cc:5495] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d49a3b4f9174b3280645f9d1ed45da0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } health_report { overall_health: UNKNOWN } } }
I20260619 14:43:16.170332 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:16.166481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46176 (local address 127.0.98.2:40441)
0619 14:43:16.166605 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:16.166613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:16.166628 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:43:16.166778 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:16.166781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:16.166864 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:16.166974 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:43:16.167264 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:16.167879 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:16.168855 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:16.169074 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:16.169174 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:16.169447 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:43:16.169922 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:43:16.170024 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:16.170140 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 14:43:16.170193 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:43:16.173203 8361 raft_consensus.cc:1261] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd [term 1 FOLLOWER]: Refusing update from remote peer 3d49a3b4f9174b3280645f9d1ed45da0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:16.173362 8081 raft_consensus.cc:1261] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3 [term 1 FOLLOWER]: Refusing update from remote peer 3d49a3b4f9174b3280645f9d1ed45da0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:16.173754 8526 consensus_queue.cc:1035] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:16.173988 8443 consensus_queue.cc:1035] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:16.185042 7924 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:43:16.185216 7924 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:43:16.187489 7924 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060: Sending DeleteTablet for 3 replicas of tablet 232cf26c62534739ab0b7e22a32064ae
I20260619 14:43:16.188190 8061 tablet_service.cc:1514] Processing DeleteTablet for tablet 232cf26c62534739ab0b7e22a32064ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36872
I20260619 14:43:16.188283 8189 tablet_service.cc:1514] Processing DeleteTablet for tablet 232cf26c62534739ab0b7e22a32064ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46170
I20260619 14:43:16.188383 8505 tablet_replica.cc:330] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3: stopping tablet replica
I20260619 14:43:16.188491 8505 raft_consensus.cc:2229] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:43:16.188485 8341 tablet_service.cc:1514] Processing DeleteTablet for tablet 232cf26c62534739ab0b7e22a32064ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43678
I20260619 14:43:16.188624 8505 raft_consensus.cc:2258] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:16.188674 8507 tablet_replica.cc:330] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd: stopping tablet replica
I20260619 14:43:16.188691 8506 tablet_replica.cc:330] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0: stopping tablet replica
I20260619 14:43:16.188798 8506 raft_consensus.cc:2229] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:43:16.188794 8507 raft_consensus.cc:2229] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:43:16.188936 8507 raft_consensus.cc:2258] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:16.189081 8506 raft_consensus.cc:2258] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:16.189410 8505 ts_tablet_manager.cc:1905] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:16.189693 8507 ts_tablet_manager.cc:1905] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:16.190009 8506 ts_tablet_manager.cc:1905] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:16.192093 8505 ts_tablet_manager.cc:1918] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:43:16.192180 8505 log.cc:1196] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-0/wal/wals/232cf26c62534739ab0b7e22a32064ae
I20260619 14:43:16.192365 8507 ts_tablet_manager.cc:1918] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:43:16.192442 8507 log.cc:1196] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/232cf26c62534739ab0b7e22a32064ae
I20260619 14:43:16.192462 8505 ts_tablet_manager.cc:1939] T 232cf26c62534739ab0b7e22a32064ae P 0882863e0497441a92182d0f4c12fbe3: Deleting consensus metadata
I20260619 14:43:16.192665 8506 ts_tablet_manager.cc:1918] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:43:16.192720 8507 ts_tablet_manager.cc:1939] T 232cf26c62534739ab0b7e22a32064ae P a7c724841ed44042a5cbfde5f5177ddd: Deleting consensus metadata
I20260619 14:43:16.192735 8506 log.cc:1196] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/232cf26c62534739ab0b7e22a32064ae
I20260619 14:43:16.193006 8506 ts_tablet_manager.cc:1939] T 232cf26c62534739ab0b7e22a32064ae P 3d49a3b4f9174b3280645f9d1ed45da0: Deleting consensus metadata
I20260619 14:43:16.193187 7909 catalog_manager.cc:4841] TS 0882863e0497441a92182d0f4c12fbe3 (127.0.98.1:35779): tablet 232cf26c62534739ab0b7e22a32064ae (table test-table [id=cc5c07eb571648838c152f0f27ae93b3]) successfully deleted
I20260619 14:43:16.193463 7908 catalog_manager.cc:4841] TS a7c724841ed44042a5cbfde5f5177ddd (127.0.98.3:35841): tablet 232cf26c62534739ab0b7e22a32064ae (table test-table [id=cc5c07eb571648838c152f0f27ae93b3]) successfully deleted
I20260619 14:43:16.193648 7910 catalog_manager.cc:4841] TS 3d49a3b4f9174b3280645f9d1ed45da0 (127.0.98.2:40441): tablet 232cf26c62534739ab0b7e22a32064ae (table test-table [id=cc5c07eb571648838c152f0f27ae93b3]) successfully deleted
I20260619 14:43:16.194208 7924 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:43:16.194690 7924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:43:16.199496 8061 tablet_service.cc:1467] Processing CreateTablet for tablet e8e92c9cf27942c6b82d3135d388a01c (DEFAULT_TABLE table=test-table [id=b67acaecb1c643a3af1f93006a41a725]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:16.199600 8189 tablet_service.cc:1467] Processing CreateTablet for tablet e8e92c9cf27942c6b82d3135d388a01c (DEFAULT_TABLE table=test-table [id=b67acaecb1c643a3af1f93006a41a725]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:16.199762 8061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8e92c9cf27942c6b82d3135d388a01c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:16.199744 8341 tablet_service.cc:1467] Processing CreateTablet for tablet e8e92c9cf27942c6b82d3135d388a01c (DEFAULT_TABLE table=test-table [id=b67acaecb1c643a3af1f93006a41a725]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:16.199873 8189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8e92c9cf27942c6b82d3135d388a01c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:16.199985 8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8e92c9cf27942c6b82d3135d388a01c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:16.202018 8520 tablet_bootstrap.cc:492] T e8e92c9cf27942c6b82d3135d388a01c P 0882863e0497441a92182d0f4c12fbe3: Bootstrap starting.
I20260619 14:43:16.202467 8521 tablet_bootstrap.cc:492] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0: Bootstrap starting.
I20260619 14:43:16.202478 8523 tablet_bootstrap.cc:492] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd: Bootstrap starting.
I20260619 14:43:16.202960 8520 tablet_bootstrap.cc:654] T e8e92c9cf27942c6b82d3135d388a01c P 0882863e0497441a92182d0f4c12fbe3: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:16.203327 8523 tablet_bootstrap.cc:654] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:16.203496 8521 tablet_bootstrap.cc:654] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:16.204190 8520 tablet_bootstrap.cc:492] T e8e92c9cf27942c6b82d3135d388a01c P 0882863e0497441a92182d0f4c12fbe3: No bootstrap required, opened a new log
I20260619 14:43:16.204274 8520 ts_tablet_manager.cc:1397] T e8e92c9cf27942c6b82d3135d388a01c P 0882863e0497441a92182d0f4c12fbe3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:43:16.204708 8520 raft_consensus.cc:348] T e8e92c9cf27942c6b82d3135d388a01c P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:16.204793 8523 tablet_bootstrap.cc:492] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd: No bootstrap required, opened a new log
I20260619 14:43:16.204839 8520 raft_consensus.cc:374] T e8e92c9cf27942c6b82d3135d388a01c P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:16.204854 8521 tablet_bootstrap.cc:492] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0: No bootstrap required, opened a new log
I20260619 14:43:16.204876 8523 ts_tablet_manager.cc:1397] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:43:16.204883 8520 raft_consensus.cc:729] T e8e92c9cf27942c6b82d3135d388a01c P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0882863e0497441a92182d0f4c12fbe3, State: Initialized, Role: FOLLOWER
I20260619 14:43:16.204928 8521 ts_tablet_manager.cc:1397] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:43:16.205014 8520 consensus_queue.cc:260] T e8e92c9cf27942c6b82d3135d388a01c P 0882863e0497441a92182d0f4c12fbe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:16.205299 8520 ts_tablet_manager.cc:1428] T e8e92c9cf27942c6b82d3135d388a01c P 0882863e0497441a92182d0f4c12fbe3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:43:16.205299 8523 raft_consensus.cc:348] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:16.205416 8523 raft_consensus.cc:374] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:16.205356 8521 raft_consensus.cc:348] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:16.205452 8523 raft_consensus.cc:729] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7c724841ed44042a5cbfde5f5177ddd, State: Initialized, Role: FOLLOWER
I20260619 14:43:16.205474 8521 raft_consensus.cc:374] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:16.205513 8521 raft_consensus.cc:729] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d49a3b4f9174b3280645f9d1ed45da0, State: Initialized, Role: FOLLOWER
I20260619 14:43:16.205590 8523 consensus_queue.cc:260] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:16.205631 8521 consensus_queue.cc:260] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:16.205858 8523 ts_tablet_manager.cc:1428] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:43:16.205907 8521 ts_tablet_manager.cc:1428] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:43:16.434445 8531 raft_consensus.cc:482] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:43:16.434640 8531 raft_consensus.cc:504] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:16.435220 8531 leader_election.cc:288] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0882863e0497441a92182d0f4c12fbe3 (127.0.98.1:35779), a7c724841ed44042a5cbfde5f5177ddd (127.0.98.3:35841)
I20260619 14:43:16.435789 8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e92c9cf27942c6b82d3135d388a01c" candidate_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7c724841ed44042a5cbfde5f5177ddd" is_pre_election: true
I20260619 14:43:16.435789 8081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e92c9cf27942c6b82d3135d388a01c" candidate_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0882863e0497441a92182d0f4c12fbe3" is_pre_election: true
I20260619 14:43:16.435953 8081 raft_consensus.cc:2454] T e8e92c9cf27942c6b82d3135d388a01c P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d49a3b4f9174b3280645f9d1ed45da0 in term 0.
I20260619 14:43:16.435950 8361 raft_consensus.cc:2454] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d49a3b4f9174b3280645f9d1ed45da0 in term 0.
I20260619 14:43:16.436264 8152 leader_election.cc:302] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0882863e0497441a92182d0f4c12fbe3, 3d49a3b4f9174b3280645f9d1ed45da0, a7c724841ed44042a5cbfde5f5177ddd; no voters:
I20260619 14:43:16.436432 8531 raft_consensus.cc:2781] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:43:16.436491 8531 raft_consensus.cc:482] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:43:16.436547 8531 raft_consensus.cc:3037] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:16.437834 8531 raft_consensus.cc:504] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:16.438189 8531 leader_election.cc:288] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [CANDIDATE]: Term 1 election: Requested vote from peers 0882863e0497441a92182d0f4c12fbe3 (127.0.98.1:35779), a7c724841ed44042a5cbfde5f5177ddd (127.0.98.3:35841)
I20260619 14:43:16.438534 8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e92c9cf27942c6b82d3135d388a01c" candidate_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7c724841ed44042a5cbfde5f5177ddd"
I20260619 14:43:16.438544 8081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e92c9cf27942c6b82d3135d388a01c" candidate_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0882863e0497441a92182d0f4c12fbe3"
I20260619 14:43:16.438638 8361 raft_consensus.cc:3037] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:16.438690 8081 raft_consensus.cc:3037] T e8e92c9cf27942c6b82d3135d388a01c P 0882863e0497441a92182d0f4c12fbe3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:16.439616 8361 raft_consensus.cc:2454] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d49a3b4f9174b3280645f9d1ed45da0 in term 1.
I20260619 14:43:16.439663 8081 raft_consensus.cc:2454] T e8e92c9cf27942c6b82d3135d388a01c P 0882863e0497441a92182d0f4c12fbe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d49a3b4f9174b3280645f9d1ed45da0 in term 1.
I20260619 14:43:16.439883 8152 leader_election.cc:302] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d49a3b4f9174b3280645f9d1ed45da0, a7c724841ed44042a5cbfde5f5177ddd; no voters:
I20260619 14:43:16.440084 8531 raft_consensus.cc:2781] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:16.440162 8531 raft_consensus.cc:686] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 LEADER]: Becoming Leader. State: Replica: 3d49a3b4f9174b3280645f9d1ed45da0, State: Running, Role: LEADER
I20260619 14:43:16.440337 8531 consensus_queue.cc:237] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } }
I20260619 14:43:16.442184 7922 catalog_manager.cc:5495] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d49a3b4f9174b3280645f9d1ed45da0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d49a3b4f9174b3280645f9d1ed45da0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } health_report { overall_health: UNKNOWN } } }
I20260619 14:43:16.514922 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:16.511152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35841 (local address 127.0.98.2:57701)
0619 14:43:16.511467 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:16.511481 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:16.511564 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:16.511790 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:16.511792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:16.511798 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:43:16.511899 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:16.511904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:16.512626 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:16.512629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:16.513270 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:16.513279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:16.513951 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:16.513956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:16.514071 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:16.514130 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:43:16.514591 (+ 461us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:43:16.514601 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 14:43:16.514688 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:43:16.514782 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":97,"threads_started":1}
I20260619 14:43:16.514923 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:16.511300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57701 (local address 127.0.98.3:35841)
0619 14:43:16.511552 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:16.511555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:16.511574 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:43:16.511645 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:16.511647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:16.511689 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:16.511763 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:43:16.511997 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:16.512515 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:16.513417 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:16.513804 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:16.513940 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:16.514332 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:43:16.514437 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:43:16.514548 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:16.514701 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 14:43:16.514755 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260619 14:43:16.520860 8361 raft_consensus.cc:1261] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd [term 1 FOLLOWER]: Refusing update from remote peer 3d49a3b4f9174b3280645f9d1ed45da0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:16.520915 8081 raft_consensus.cc:1261] T e8e92c9cf27942c6b82d3135d388a01c P 0882863e0497441a92182d0f4c12fbe3 [term 1 FOLLOWER]: Refusing update from remote peer 3d49a3b4f9174b3280645f9d1ed45da0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:16.521297 8531 consensus_queue.cc:1035] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0882863e0497441a92182d0f4c12fbe3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:16.521445 8443 consensus_queue.cc:1035] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c724841ed44042a5cbfde5f5177ddd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:16.538273 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:16.535024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54227 (local address 127.0.98.2:40441)
0619 14:43:16.535135 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:16.535139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:16.535151 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:43:16.535190 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:16.535199 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:16.535238 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:16.535324 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:43:16.535621 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:16.536358 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:16.537195 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:16.537457 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:16.537550 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:16.537828 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:43:16.537908 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:43:16.537988 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:16.538081 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 14:43:16.538131 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:43:16.538275 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:16.534632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40441 (local address 127.0.98.3:54227)
0619 14:43:16.534931 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:16.534949 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:16.535049 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:16.535345 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:16.535347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:16.535355 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:43:16.535492 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:16.535501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:16.536476 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:16.536478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:16.537085 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:16.537092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:16.537572 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:16.537577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:16.537672 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:16.537699 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:43:16.537991 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:43:16.537999 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:43:16.538061 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 14:43:16.538148 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":129,"threads_started":1}
I20260619 14:43:16.564576 7923 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:43:16.564742 7923 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:43:16.567024 7923 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c02054cd162d4fbda16a00a5c1525060: Sending DeleteTablet for 3 replicas of tablet e8e92c9cf27942c6b82d3135d388a01c
I20260619 14:43:16.567647 8061 tablet_service.cc:1514] Processing DeleteTablet for tablet e8e92c9cf27942c6b82d3135d388a01c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36872
I20260619 14:43:16.567795 8341 tablet_service.cc:1514] Processing DeleteTablet for tablet e8e92c9cf27942c6b82d3135d388a01c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43678
I20260619 14:43:16.567839 8505 tablet_replica.cc:330] T e8e92c9cf27942c6b82d3135d388a01c P 0882863e0497441a92182d0f4c12fbe3: stopping tablet replica
I20260619 14:43:16.567963 8507 tablet_replica.cc:330] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd: stopping tablet replica
I20260619 14:43:16.567958 8505 raft_consensus.cc:2229] T e8e92c9cf27942c6b82d3135d388a01c P 0882863e0497441a92182d0f4c12fbe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:43:16.568074 8505 raft_consensus.cc:2258] T e8e92c9cf27942c6b82d3135d388a01c P 0882863e0497441a92182d0f4c12fbe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:16.568066 8507 raft_consensus.cc:2229] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:43:16.568173 8507 raft_consensus.cc:2258] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:16.568295 8189 tablet_service.cc:1514] Processing DeleteTablet for tablet e8e92c9cf27942c6b82d3135d388a01c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46170
I20260619 14:43:16.568468 8506 tablet_replica.cc:330] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0: stopping tablet replica
I20260619 14:43:16.568603 8506 raft_consensus.cc:2229] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:43:16.568800 8506 raft_consensus.cc:2258] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:16.568960 8505 ts_tablet_manager.cc:1905] T e8e92c9cf27942c6b82d3135d388a01c P 0882863e0497441a92182d0f4c12fbe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:16.568965 8507 ts_tablet_manager.cc:1905] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:16.569131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 7996
I20260619 14:43:16.569538 8506 ts_tablet_manager.cc:1905] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:16.571099 8507 ts_tablet_manager.cc:1918] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:43:16.571153 8507 log.cc:1196] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/e8e92c9cf27942c6b82d3135d388a01c
I20260619 14:43:16.571337 8507 ts_tablet_manager.cc:1939] T e8e92c9cf27942c6b82d3135d388a01c P a7c724841ed44042a5cbfde5f5177ddd: Deleting consensus metadata
I20260619 14:43:16.572886 7908 catalog_manager.cc:4841] TS a7c724841ed44042a5cbfde5f5177ddd (127.0.98.3:35841): tablet e8e92c9cf27942c6b82d3135d388a01c (table test-table [id=b67acaecb1c643a3af1f93006a41a725]) successfully deleted
I20260619 14:43:16.573062 8506 ts_tablet_manager.cc:1918] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:43:16.573138 8506 log.cc:1196] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/e8e92c9cf27942c6b82d3135d388a01c
I20260619 14:43:16.573364 8506 ts_tablet_manager.cc:1939] T e8e92c9cf27942c6b82d3135d388a01c P 3d49a3b4f9174b3280645f9d1ed45da0: Deleting consensus metadata
I20260619 14:43:16.573977 7910 catalog_manager.cc:4841] TS 3d49a3b4f9174b3280645f9d1ed45da0 (127.0.98.2:40441): tablet e8e92c9cf27942c6b82d3135d388a01c (table test-table [id=b67acaecb1c643a3af1f93006a41a725]) successfully deleted
W20260619 14:43:16.578399 7909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35779 (error 108)
I20260619 14:43:16.578488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 8136
I20260619 14:43:16.580791 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:16.579939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35779 (local address 127.0.0.1:36884)
0619 14:43:16.580578 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:16.580661 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":59,"threads_started":1}
W20260619 14:43:16.580966 7909 catalog_manager.cc:4584] TS 0882863e0497441a92182d0f4c12fbe3 (127.0.98.1:35779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e8e92c9cf27942c6b82d3135d388a01c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35779: connect: Connection refused (error 111)
I20260619 14:43:16.586319 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 8276
I20260619 14:43:16.593145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 7892
2026-06-19T14:43:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3738 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:43:16 dist-test-slave-cr22 krb5kdc[8552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:43:16 dist-test-slave-cr22 krb5kdc[8552](info): set up 2 sockets
Jun 19 14:43:16 dist-test-slave-cr22 krb5kdc[8552](info): commencing operation
krb5kdc: starting...
W20260619 14:43:17.688338 392 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:43:17 dist-test-slave-cr22 krb5kdc[8552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:43:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:43:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:43:17.823318 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34847
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:17.939879 8565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:17.940155 8565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:17.940196 8565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:17.944298 8565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:43:17.944360 8565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:17.944379 8565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:17.944396 8565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:43:17.944434 8565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:43:17.949915 8565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:17.951037 8565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:17.952564 8565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:17.962189 8570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:17.962193 8573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:17.962440 8565 server_base.cc:1034] running on GCE node
W20260619 14:43:17.962210 8571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:17.963052 8565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:17.964206 8565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:17.965411 8565 hybrid_clock.cc:648] HybridClock initialized: now 1781880197965397 us; error 29 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T14:43:17Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1387 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:43:18 dist-test-slave-cr22 krb5kdc[8578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:43:18 dist-test-slave-cr22 krb5kdc[8578](info): set up 2 sockets
Jun 19 14:43:18 dist-test-slave-cr22 krb5kdc[8578](info): commencing operation
krb5kdc: starting...
W20260619 14:43:19.048292 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:43:19 dist-test-slave-cr22 krb5kdc[8578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:43:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:43:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:43:19.172328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44033
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:19.281821 8591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:19.282141 8591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:19.282214 8591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:19.286286 8591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:43:19.286397 8591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:19.286437 8591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:19.286496 8591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:43:19.286535 8591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:43:19.291939 8591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:19.293087 8591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:19.294150 8591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:19.302827 8597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:19.303052 8591 server_base.cc:1034] running on GCE node
W20260619 14:43:19.302850 8599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:19.302925 8596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:19.303820 8591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:19.304944 8591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:19.306159 8591 hybrid_clock.cc:648] HybridClock initialized: now 1781880199306139 us; error 34 us; skew 500 ppm
Jun 19 14:43:19 dist-test-slave-cr22 krb5kdc[8578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:19.309095 8591 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:43:19.309950 8591 webserver.cc:460] Webserver started at http://127.0.98.62:38669/ using document root <none> and password file <none>
I20260619 14:43:19.310472 8591 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:19.310529 8591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:19.310700 8591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:19.313143 8591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "03efd39fa3dd4a029c167322375aee64"
format_stamp: "Formatted at 2026-06-19 14:43:19 on dist-test-slave-cr22"
server_key: "ff74f2ca30046af2119c084dc3b9aed2"
server_key_iv: "397c75be950e096aca6db7898f663887"
server_key_version: "encryptionkey@0"
I20260619 14:43:19.313652 8591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "03efd39fa3dd4a029c167322375aee64"
format_stamp: "Formatted at 2026-06-19 14:43:19 on dist-test-slave-cr22"
server_key: "ff74f2ca30046af2119c084dc3b9aed2"
server_key_iv: "397c75be950e096aca6db7898f663887"
server_key_version: "encryptionkey@0"
I20260619 14:43:19.317309 8591 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:43:19.319752 8606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:19.321005 8591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:43:19.321193 8591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "03efd39fa3dd4a029c167322375aee64"
format_stamp: "Formatted at 2026-06-19 14:43:19 on dist-test-slave-cr22"
server_key: "ff74f2ca30046af2119c084dc3b9aed2"
server_key_iv: "397c75be950e096aca6db7898f663887"
server_key_version: "encryptionkey@0"
I20260619 14:43:19.321300 8591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:19.347595 8591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:19.351070 8591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:19.351313 8591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:19.359000 8591 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44033
I20260619 14:43:19.359027 8658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44033 every 8 connection(s)
I20260619 14:43:19.360031 8591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:43:19.362722 8659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:19.368183 8659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64: Bootstrap starting.
I20260619 14:43:19.368395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 8591
I20260619 14:43:19.368527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:43:19.368814 392 external_mini_cluster.cc:1428] Setting key d55ed8e01a2e40d83bb62267e99384f8
I20260619 14:43:19.370821 8659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:19.371600 8659 log.cc:824] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:19.373683 8659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64: No bootstrap required, opened a new log
Jun 19 14:43:19 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:43:19.378263 8659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03efd39fa3dd4a029c167322375aee64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } }
I20260619 14:43:19.378489 8659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:19.378552 8659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03efd39fa3dd4a029c167322375aee64, State: Initialized, Role: FOLLOWER
I20260619 14:43:19.379011 8659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03efd39fa3dd4a029c167322375aee64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } }
I20260619 14:43:19.379181 8659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:43:19.379244 8659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:43:19.379495 8659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:19.380575 8659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03efd39fa3dd4a029c167322375aee64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } }
I20260619 14:43:19.380940 8659 leader_election.cc:302] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03efd39fa3dd4a029c167322375aee64; no voters:
I20260619 14:43:19.381291 8659 leader_election.cc:288] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:43:19.381402 8664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:19.381577 8664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [term 1 LEADER]: Becoming Leader. State: Replica: 03efd39fa3dd4a029c167322375aee64, State: Running, Role: LEADER
I20260619 14:43:19.381870 8664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03efd39fa3dd4a029c167322375aee64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } }
I20260619 14:43:19.382606 8659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:43:19.383062 8666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03efd39fa3dd4a029c167322375aee64. Latest consensus state: current_term: 1 leader_uuid: "03efd39fa3dd4a029c167322375aee64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03efd39fa3dd4a029c167322375aee64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } } }
I20260619 14:43:19.383203 8666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:19.383667 8668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:43:19.383973 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:19.370198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39780 (local address 127.0.98.62:44033)
0619 14:43:19.370614 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:19.370627 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:19.370675 (+ 48us) server_negotiation.cc:407] Connection header received
0619 14:43:19.371355 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:19.371372 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:19.371820 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:19.372168 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:19.372961 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:19.374033 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:19.375186 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:19.375697 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:19.375818 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:19.378684 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:19.378689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:19.378702 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:19.378729 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:19.380572 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:19.381094 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:19.381105 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:19.381110 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:19.381243 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:19.381545 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:19.381548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:19.381552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:19.382078 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:19.382268 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:19.382631 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 14:43:19.382946 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":116,"threads_started":1}
I20260619 14:43:19.383616 8665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03efd39fa3dd4a029c167322375aee64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03efd39fa3dd4a029c167322375aee64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } } }
I20260619 14:43:19.384722 8665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:19.386969 8668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:43:19.391849 8668 catalog_manager.cc:1347] Generated new cluster ID: 814ab429fc334d64bc6d890e56db5cdc
I20260619 14:43:19.391907 8668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:43:19.439234 8668 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:43:19.440536 8668 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:43:19.446532 8668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64: Generated new TSK 0
I20260619 14:43:19.447240 8668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:43:19.479628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44033
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:19.595739 8685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:19.596048 8685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:19.596133 8685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:19.600332 8685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:19.600431 8685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:19.600524 8685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:43:19.605873 8685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:19.606953 8685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:19.608052 8685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:19.617997 8690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:19.618018 8693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:19.618032 8691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:19.618399 8685 server_base.cc:1034] running on GCE node
I20260619 14:43:19.618798 8685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:19.619422 8685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:19.620692 8685 hybrid_clock.cc:648] HybridClock initialized: now 1781880199620660 us; error 40 us; skew 500 ppm
Jun 19 14:43:19 dist-test-slave-cr22 krb5kdc[8578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:19.623812 8685 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:43:19.624742 8685 webserver.cc:460] Webserver started at http://127.0.98.1:32861/ using document root <none> and password file <none>
I20260619 14:43:19.625268 8685 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:19.625325 8685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:19.625818 8685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:19.627923 8685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "8b37fcec107f4c7b8c3a191e1920d29e"
format_stamp: "Formatted at 2026-06-19 14:43:19 on dist-test-slave-cr22"
server_key: "db385dbc7b7d9fffb0baa0d93a07def9"
server_key_iv: "6563ef0ba33f580c02e8878c378588b6"
server_key_version: "encryptionkey@0"
I20260619 14:43:19.628412 8685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8b37fcec107f4c7b8c3a191e1920d29e"
format_stamp: "Formatted at 2026-06-19 14:43:19 on dist-test-slave-cr22"
server_key: "db385dbc7b7d9fffb0baa0d93a07def9"
server_key_iv: "6563ef0ba33f580c02e8878c378588b6"
server_key_version: "encryptionkey@0"
I20260619 14:43:19.632155 8685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:43:19.634617 8700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:19.635979 8685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:43:19.636083 8685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "8b37fcec107f4c7b8c3a191e1920d29e"
format_stamp: "Formatted at 2026-06-19 14:43:19 on dist-test-slave-cr22"
server_key: "db385dbc7b7d9fffb0baa0d93a07def9"
server_key_iv: "6563ef0ba33f580c02e8878c378588b6"
server_key_version: "encryptionkey@0"
I20260619 14:43:19.636171 8685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:19.651295 8685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:19.654660 8685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:19.654819 8685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:19.655349 8685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:19.656375 8685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:19.656430 8685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:19.656474 8685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:19.656520 8685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:19.665269 8685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44963
I20260619 14:43:19.665331 8813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44963 every 8 connection(s)
I20260619 14:43:19.666178 8685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:43:19.667738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 8685
I20260619 14:43:19.667878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:43:19.668227 392 external_mini_cluster.cc:1428] Setting key f11277965157b5d59a908af3102df4d3
Jun 19 14:43:19 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:43:19.681377 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:19.668247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54103 (local address 127.0.98.62:44033)
0619 14:43:19.668388 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:19.668394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:19.669332 (+ 938us) server_negotiation.cc:407] Connection header received
0619 14:43:19.670835 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:19.670838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:19.670888 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:19.670972 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:19.672216 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:19.672739 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:19.674564 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:19.674789 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:19.674874 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:19.677889 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:19.677892 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:19.677895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:19.677916 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:19.679355 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:19.679878 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:19.679882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:19.679884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:19.679962 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:19.680311 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:19.680318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:19.680320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:19.680485 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:19.680589 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:19.681071 (+ 482us) server_negotiation.cc:299] Negotiation successful
0619 14:43:19.681194 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:43:19.682245 8816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:19.668647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:54103)
0619 14:43:19.669179 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:19.669241 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:19.670616 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:19.671156 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:19.671162 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:19.671591 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:19.672068 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:19.672081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:19.672867 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:19.672873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:19.674424 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:19.674438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:19.674953 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:19.674960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:19.675206 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:19.675980 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:19.676004 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:19.677702 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:19.679495 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:19.679500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:19.679514 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:19.679767 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:19.680061 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:19.680064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:19.680066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:19.680201 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:19.680587 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:19.680593 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:19.680840 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 14:43:19.681017 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 14:43:19.681331 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":148,"threads_started":1}
I20260619 14:43:19.683648 8814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44033
I20260619 14:43:19.683960 8814 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:19.684485 8814 heartbeater.cc:511] Master 127.0.98.62:44033 requested a full tablet report, sending...
I20260619 14:43:19.686151 8623 ts_manager.cc:194] Registered new tserver with Master: 8b37fcec107f4c7b8c3a191e1920d29e (127.0.98.1:44963)
I20260619 14:43:19.687808 8623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:43:19.699476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44033
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:19.809957 8819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:19.810209 8819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:19.810250 8819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:19.814365 8819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:19.814419 8819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:19.814509 8819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:43:19.819716 8819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:19.820726 8819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:19.821708 8819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:19.831310 8824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:19.831418 8825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:19.831653 8819 server_base.cc:1034] running on GCE node
W20260619 14:43:19.831362 8827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:19.832136 8819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:19.832816 8819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:19.834012 8819 hybrid_clock.cc:648] HybridClock initialized: now 1781880199833984 us; error 57 us; skew 500 ppm
Jun 19 14:43:19 dist-test-slave-cr22 krb5kdc[8578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:19.837049 8819 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:43:19.838043 8819 webserver.cc:460] Webserver started at http://127.0.98.2:43991/ using document root <none> and password file <none>
I20260619 14:43:19.838627 8819 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:19.838696 8819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:19.838923 8819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:19.841344 8819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "3ca29106969f4dd98f57b141a2982967"
format_stamp: "Formatted at 2026-06-19 14:43:19 on dist-test-slave-cr22"
server_key: "06c491ef275f95181b226a90bb108ef1"
server_key_iv: "2ffafd5823b8782f193f6ee577104d2d"
server_key_version: "encryptionkey@0"
I20260619 14:43:19.842041 8819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3ca29106969f4dd98f57b141a2982967"
format_stamp: "Formatted at 2026-06-19 14:43:19 on dist-test-slave-cr22"
server_key: "06c491ef275f95181b226a90bb108ef1"
server_key_iv: "2ffafd5823b8782f193f6ee577104d2d"
server_key_version: "encryptionkey@0"
I20260619 14:43:19.846112 8819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 14:43:19.848830 8834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:19.850265 8819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:43:19.850400 8819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "3ca29106969f4dd98f57b141a2982967"
format_stamp: "Formatted at 2026-06-19 14:43:19 on dist-test-slave-cr22"
server_key: "06c491ef275f95181b226a90bb108ef1"
server_key_iv: "2ffafd5823b8782f193f6ee577104d2d"
server_key_version: "encryptionkey@0"
I20260619 14:43:19.850518 8819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:19.869774 8819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:19.873396 8819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:19.873610 8819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:19.874291 8819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:19.875137 8819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:19.875185 8819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:19.875252 8819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:19.875283 8819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:19.885274 8819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35861
I20260619 14:43:19.885304 8947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35861 every 8 connection(s)
I20260619 14:43:19.886263 8819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 14:43:19 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:43:19.896370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 8819
I20260619 14:43:19.896524 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:43:19.896904 392 external_mini_cluster.cc:1428] Setting key 2ceebbc50d75bf32310840ba913aa4db
I20260619 14:43:19.900441 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:19.888130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38721 (local address 127.0.98.62:44033)
0619 14:43:19.888267 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:19.888272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:19.889061 (+ 789us) server_negotiation.cc:407] Connection header received
0619 14:43:19.890460 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:19.890464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:19.890521 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:19.890621 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:19.891805 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:19.892327 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:19.893425 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:19.893647 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:19.893754 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:19.896226 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:19.896230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:19.896231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:19.896254 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:19.898056 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:19.898648 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:19.898652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:19.898654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:19.898755 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:19.899084 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:19.899091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:19.899096 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:19.899417 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:19.899565 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:19.900122 (+ 557us) server_negotiation.cc:299] Negotiation successful
0619 14:43:19.900274 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:43:19.901541 8950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:19.888511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:38721)
0619 14:43:19.888920 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:19.888961 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:19.890231 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:19.890784 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:19.890791 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:19.891134 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:19.891663 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:19.891677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:19.892445 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:19.892449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:19.893302 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:19.893311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:19.893792 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:19.893796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:19.893966 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:19.894569 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:19.894585 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:19.896038 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:19.898175 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:19.898180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:19.898194 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:19.898543 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:19.898860 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:19.898862 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:19.898864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:19.898987 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:19.899585 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:19.899590 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:19.899824 (+ 234us) client_negotiation.cc:769] Sending connection context
0619 14:43:19.900035 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 14:43:19.900398 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":94,"threads_started":1}
I20260619 14:43:19.902916 8948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44033
I20260619 14:43:19.903200 8948 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:19.903841 8948 heartbeater.cc:511] Master 127.0.98.62:44033 requested a full tablet report, sending...
I20260619 14:43:19.905059 8623 ts_manager.cc:194] Registered new tserver with Master: 3ca29106969f4dd98f57b141a2982967 (127.0.98.2:35861)
I20260619 14:43:19.905742 8623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38721
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:43:19.927587 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44033
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:20.043579 8953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:20.043883 8953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:20.043933 8953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:20.048621 8953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:20.048779 8953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:20.048846 8953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:43:20.054817 8953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:20.055953 8953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:20.057013 8953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:43:20.067512 8953 server_base.cc:1034] running on GCE node
W20260619 14:43:20.067422 8961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:20.067391 8958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:20.067425 8959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:20.068233 8953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:20.068975 8953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:20.070188 8953 hybrid_clock.cc:648] HybridClock initialized: now 1781880200070154 us; error 55 us; skew 500 ppm
Jun 19 14:43:20 dist-test-slave-cr22 krb5kdc[8578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:20.073690 8953 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:43:20.074760 8953 webserver.cc:460] Webserver started at http://127.0.98.3:42037/ using document root <none> and password file <none>
I20260619 14:43:20.075376 8953 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:20.075511 8953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:20.075767 8953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:20.077984 8953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "8a51fff4d77f466a84a18dc371bbb534"
format_stamp: "Formatted at 2026-06-19 14:43:20 on dist-test-slave-cr22"
server_key: "bfb05eec4cff1a374466bb7ca17591a4"
server_key_iv: "ec80ba71f678dc2020090c64ef785934"
server_key_version: "encryptionkey@0"
I20260619 14:43:20.078593 8953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8a51fff4d77f466a84a18dc371bbb534"
format_stamp: "Formatted at 2026-06-19 14:43:20 on dist-test-slave-cr22"
server_key: "bfb05eec4cff1a374466bb7ca17591a4"
server_key_iv: "ec80ba71f678dc2020090c64ef785934"
server_key_version: "encryptionkey@0"
I20260619 14:43:20.082661 8953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 14:43:20.085587 8968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:20.087004 8953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:43:20.087169 8953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "8a51fff4d77f466a84a18dc371bbb534"
format_stamp: "Formatted at 2026-06-19 14:43:20 on dist-test-slave-cr22"
server_key: "bfb05eec4cff1a374466bb7ca17591a4"
server_key_iv: "ec80ba71f678dc2020090c64ef785934"
server_key_version: "encryptionkey@0"
I20260619 14:43:20.087318 8953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:20.105902 8953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:20.109735 8953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:20.109982 8953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:20.110661 8953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:20.111622 8953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:20.111696 8953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:20.111768 8953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:20.111821 8953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:20.121299 8953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37481
I20260619 14:43:20.121333 9081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37481 every 8 connection(s)
I20260619 14:43:20.122256 8953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:43:20.124027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 8953
I20260619 14:43:20.124184 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:43:20.124507 392 external_mini_cluster.cc:1428] Setting key 959a74c666d5301d6e4c91568b5fbb8e
Jun 19 14:43:20 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:43:20.138087 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:20.124415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49031 (local address 127.0.98.62:44033)
0619 14:43:20.124585 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:20.124590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:20.125695 (+ 1105us) server_negotiation.cc:407] Connection header received
0619 14:43:20.127258 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:20.127261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:20.127312 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:20.127388 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:20.128554 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.129091 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.130238 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.130465 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.130574 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:20.133231 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:20.133234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:20.133236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:20.133260 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:20.135083 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:20.135862 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:20.135865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:20.135866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:20.135942 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:20.136450 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:20.136453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:20.136454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:20.136757 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:20.136854 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:20.137662 (+ 808us) server_negotiation.cc:299] Negotiation successful
0619 14:43:20.137913 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:43:20.139081 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:20.124980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.3:49031)
0619 14:43:20.125551 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:20.125604 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:20.127059 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:20.127583 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:20.127589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:20.127935 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:20.128412 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:20.128424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.129214 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.129218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:20.130095 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:20.130103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.130607 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.130611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:20.130786 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:20.131396 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:20.131412 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:20.133056 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:20.135230 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:20.135236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:20.135248 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:20.135713 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:20.136077 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:20.136080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:20.136081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:20.136236 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:20.136945 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:20.136952 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:20.137313 (+ 361us) client_negotiation.cc:769] Sending connection context
0619 14:43:20.137604 (+ 291us) client_negotiation.cc:240] Negotiation successful
0619 14:43:20.138054 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":162,"threads_started":1}
I20260619 14:43:20.140240 9082 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44033
I20260619 14:43:20.140493 9082 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:20.140998 9082 heartbeater.cc:511] Master 127.0.98.62:44033 requested a full tablet report, sending...
I20260619 14:43:20.142072 8623 ts_manager.cc:194] Registered new tserver with Master: 8a51fff4d77f466a84a18dc371bbb534 (127.0.98.3:37481)
I20260619 14:43:20.142802 8623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49031
I20260619 14:43:20.150955 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 14:43:20.231361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 8685
I20260619 14:43:20.237200 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 8819
I20260619 14:43:20.242312 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 8953
I20260619 14:43:20.248286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 8591
I20260619 14:43:20.257905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44033
--webserver_interface=127.0.98.62
--webserver_port=38669
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:20.367314 9091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:20.367676 9091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:20.367722 9091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:20.371747 9091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:43:20.371804 9091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:20.371824 9091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:20.371840 9091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:43:20.371855 9091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:43:20.377291 9091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:20.378341 9091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:20.379366 9091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:20.387813 9097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:20.387820 9099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:20.387820 9096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:20.388041 9091 server_base.cc:1034] running on GCE node
I20260619 14:43:20.388587 9091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:20.389678 9091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:20.390877 9091 hybrid_clock.cc:648] HybridClock initialized: now 1781880200390864 us; error 32 us; skew 500 ppm
Jun 19 14:43:20 dist-test-slave-cr22 krb5kdc[8578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:20.393682 9091 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:43:20.394589 9091 webserver.cc:460] Webserver started at http://127.0.98.62:38669/ using document root <none> and password file <none>
I20260619 14:43:20.395161 9091 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:20.395282 9091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:20.399106 9091 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:43:20.401855 9106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:20.403139 9091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:43:20.403395 9091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "03efd39fa3dd4a029c167322375aee64"
format_stamp: "Formatted at 2026-06-19 14:43:19 on dist-test-slave-cr22"
server_key: "ff74f2ca30046af2119c084dc3b9aed2"
server_key_iv: "397c75be950e096aca6db7898f663887"
server_key_version: "encryptionkey@0"
I20260619 14:43:20.404031 9091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:20.432924 9091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:20.436160 9091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:20.436370 9091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:20.443593 9091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44033
I20260619 14:43:20.443619 9158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44033 every 8 connection(s)
I20260619 14:43:20.444664 9091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:43:20.448935 9159 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:43:20.450525 9159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64: Bootstrap starting.
I20260619 14:43:20.454527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 9091
I20260619 14:43:20.455001 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44963
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44033
--webserver_port=32861
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:43:20.467908 9159 log.cc:824] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:20.476289 9159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:43:20.476776 9159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64: Bootstrap complete.
I20260619 14:43:20.482900 9159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03efd39fa3dd4a029c167322375aee64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } }
I20260619 14:43:20.483420 9159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03efd39fa3dd4a029c167322375aee64, State: Initialized, Role: FOLLOWER
I20260619 14:43:20.483995 9159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03efd39fa3dd4a029c167322375aee64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } }
I20260619 14:43:20.484140 9159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:43:20.484211 9159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:43:20.484308 9159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:43:20.486869 9159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03efd39fa3dd4a029c167322375aee64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } }
I20260619 14:43:20.487223 9159 leader_election.cc:302] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03efd39fa3dd4a029c167322375aee64; no voters:
I20260619 14:43:20.487576 9159 leader_election.cc:288] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:43:20.487726 9163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:43:20.487994 9163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [term 2 LEADER]: Becoming Leader. State: Replica: 03efd39fa3dd4a029c167322375aee64, State: Running, Role: LEADER
I20260619 14:43:20.488344 9163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03efd39fa3dd4a029c167322375aee64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } }
I20260619 14:43:20.488837 9159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:43:20.489787 9164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "03efd39fa3dd4a029c167322375aee64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03efd39fa3dd4a029c167322375aee64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } } }
I20260619 14:43:20.489910 9164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:20.490162 9165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03efd39fa3dd4a029c167322375aee64. Latest consensus state: current_term: 2 leader_uuid: "03efd39fa3dd4a029c167322375aee64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03efd39fa3dd4a029c167322375aee64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } } }
I20260619 14:43:20.490257 9165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:20.490854 9172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:43:20.493534 9172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:43:20.494120 9172 catalog_manager.cc:1259] Loaded cluster ID: 814ab429fc334d64bc6d890e56db5cdc
I20260619 14:43:20.494197 9172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:43:20.496017 9172 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:43:20.496613 9172 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 03efd39fa3dd4a029c167322375aee64: Loaded TSK: 0
I20260619 14:43:20.497551 9172 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:43:20.588856 9161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:20.589207 9161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:20.589299 9161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:20.594420 9161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:20.594511 9161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:20.594615 9161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:43:20.601198 9161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=32861
--tserver_master_addrs=127.0.98.62:44033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:20.602267 9161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:20.603293 9161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:20.614002 9185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:20.614187 9186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:20.614238 9188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:20.614545 9161 server_base.cc:1034] running on GCE node
I20260619 14:43:20.615195 9161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:20.615904 9161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:20.617091 9161 hybrid_clock.cc:648] HybridClock initialized: now 1781880200617066 us; error 44 us; skew 500 ppm
Jun 19 14:43:20 dist-test-slave-cr22 krb5kdc[8578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880200, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:20.623014 9161 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 14:43:20.624115 9161 webserver.cc:460] Webserver started at http://127.0.98.1:32861/ using document root <none> and password file <none>
I20260619 14:43:20.624703 9161 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:20.624780 9161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:20.629257 9161 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:43:20.631507 9195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:20.632900 9161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:43:20.633003 9161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "8b37fcec107f4c7b8c3a191e1920d29e"
format_stamp: "Formatted at 2026-06-19 14:43:19 on dist-test-slave-cr22"
server_key: "db385dbc7b7d9fffb0baa0d93a07def9"
server_key_iv: "6563ef0ba33f580c02e8878c378588b6"
server_key_version: "encryptionkey@0"
I20260619 14:43:20.633591 9161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:20.648054 9161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:20.651371 9161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:20.651604 9161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:20.652261 9161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:20.653092 9161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:20.653137 9161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:20.653198 9161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:20.653230 9161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:20.664464 9161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44963
I20260619 14:43:20.664680 9308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44963 every 8 connection(s)
I20260619 14:43:20.665922 9161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:43:20.672344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 9161
I20260619 14:43:20.673038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35861
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44033
--webserver_port=43991
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:43:20 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:20 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:20.684003 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:20.668875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:56149)
0619 14:43:20.669449 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:20.669509 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:20.672131 (+ 2622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:20.673648 (+ 1517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:20.673661 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:20.674367 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:20.675101 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:20.675139 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.676884 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.676891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:20.678059 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:20.678075 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.679018 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.679035 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:20.679235 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:20.679952 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:20.679970 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:20.682901 (+ 2931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":143,"threads_started":1}
W20260619 14:43:20.683995 9311 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:20.668822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56149 (local address 127.0.98.62:44033)
0619 14:43:20.669381 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:20.669401 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:20.669661 (+ 260us) server_negotiation.cc:407] Connection header received
0619 14:43:20.672666 (+ 3005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:20.672681 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:20.673150 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:20.673512 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:20.675338 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.676734 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.678254 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.678836 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.679029 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:20.683153 (+ 4124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56149: BlockingRecv error: recv got EOF from 127.0.98.1:56149 (error 108)
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":141,"threads_started":1}
W20260619 14:43:20.684911 9309 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44033 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44033: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 14:43:20 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:20 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:20.692862 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:20.685435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:57101)
0619 14:43:20.685620 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:20.685641 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:20.685746 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:20.686072 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:20.686075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:20.686267 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:20.686409 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:20.686422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.687550 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.687555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:20.688918 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:20.688931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.689648 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.689653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:20.689776 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:20.690437 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:20.690457 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:20.692711 (+ 2254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 14:43:20.693274 9311 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:20.685715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57101 (local address 127.0.98.62:44033)
0619 14:43:20.685848 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:20.685853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:20.685866 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:43:20.685916 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:20.685918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:20.685959 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:20.686056 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:20.686559 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.687391 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.689133 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.689506 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.689609 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:20.693169 (+ 3560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57101: BlockingRecv error: recv got EOF from 127.0.98.1:57101 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 14:43:20 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:20 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:20.701289 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:20.695373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:59909)
0619 14:43:20.695637 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:20.695650 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:20.695724 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:20.696127 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:20.696130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:20.696351 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:20.696466 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:20.696472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.697289 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.697296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:20.697965 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:20.697976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.698483 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.698487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:20.698572 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:20.699126 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:20.699142 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:20.701124 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192}
W20260619 14:43:20.701397 9311 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:20.695654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59909 (local address 127.0.98.62:44033)
0619 14:43:20.695783 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:20.695788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:20.695803 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:43:20.695848 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:20.695851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:20.695897 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:20.695991 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:20.696602 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.697168 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.698098 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.698376 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.698482 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:20.701262 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59909: BlockingRecv error: recv got EOF from 127.0.98.1:59909 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 14:43:20.701853 9309 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:43:20.798035 9313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:20.798286 9313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:20.798331 9313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:20.802778 9313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:20.802845 9313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:20.802906 9313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:43:20.808418 9313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35861
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43991
--tserver_master_addrs=127.0.98.62:44033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:20.809559 9313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:20.810526 9313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:20.819998 9319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:20.820061 9321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:20.820056 9313 server_base.cc:1034] running on GCE node
W20260619 14:43:20.820004 9318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:20.820590 9313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:20.821139 9313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:20.822294 9313 hybrid_clock.cc:648] HybridClock initialized: now 1781880200822267 us; error 42 us; skew 500 ppm
Jun 19 14:43:20 dist-test-slave-cr22 krb5kdc[8578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880200, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:20.825019 9313 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 14:43:20.825819 9313 webserver.cc:460] Webserver started at http://127.0.98.2:43991/ using document root <none> and password file <none>
I20260619 14:43:20.826308 9313 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:20.826372 9313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:20.830088 9313 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260619 14:43:20.832146 9328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:20.833415 9313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:43:20.833570 9313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "3ca29106969f4dd98f57b141a2982967"
format_stamp: "Formatted at 2026-06-19 14:43:19 on dist-test-slave-cr22"
server_key: "06c491ef275f95181b226a90bb108ef1"
server_key_iv: "2ffafd5823b8782f193f6ee577104d2d"
server_key_version: "encryptionkey@0"
I20260619 14:43:20.834153 9313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:20.845952 9313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:20.849181 9313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:20.849386 9313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:20.849962 9313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:20.850791 9313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:20.850867 9313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:20.850935 9313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:20.850992 9313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:20.860376 9313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35861
I20260619 14:43:20.860397 9441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35861 every 8 connection(s)
I20260619 14:43:20.861363 9313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 14:43:20 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:20.870985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 9313
Jun 19 14:43:20 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:20.871495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37481
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44033
--webserver_port=42037
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:20.872828 9311 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:20.863161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44373 (local address 127.0.98.62:44033)
0619 14:43:20.863330 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:20.863335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:20.864300 (+ 965us) server_negotiation.cc:407] Connection header received
0619 14:43:20.865607 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:20.865611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:20.865660 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:20.865736 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:20.866875 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.867400 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.868675 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.868920 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.869006 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:20.872716 (+ 3710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44373: BlockingRecv error: recv got EOF from 127.0.98.2:44373 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 14:43:20.873492 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:20.863657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:44373)
0619 14:43:20.864147 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:20.864187 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:20.865398 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:20.865925 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:20.865931 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:20.866272 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:20.866718 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:20.866731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.867590 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.867593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:20.868557 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:20.868566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.869096 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.869099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:20.869281 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:20.869856 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:20.869872 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:20.872616 (+ 2744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":104,"threads_started":1}
W20260619 14:43:20.874084 9442 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44033 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44033: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 14:43:20 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:20 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:20.880424 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:20.874552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:43575)
0619 14:43:20.874730 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:20.874744 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:20.874821 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:20.875066 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:20.875068 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:20.875201 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:20.875289 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:20.875293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.876415 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.876418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:20.877031 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:20.877039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.877721 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.877724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:20.877794 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:20.878314 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:20.878325 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:20.880327 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99}
W20260619 14:43:20.880654 9311 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:20.874671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43575 (local address 127.0.98.62:44033)
0619 14:43:20.874793 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:20.874799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:20.874816 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:43:20.874907 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:20.874912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:20.874967 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:20.875044 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:20.875408 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.876286 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.877150 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.877564 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.877688 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:20.880504 (+ 2816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43575: BlockingRecv error: recv got EOF from 127.0.98.2:43575 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 19 14:43:20 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:20 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:20.887318 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:20.881117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:48023)
0619 14:43:20.881204 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:20.881218 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:20.881291 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:20.881581 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:20.881582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:20.881757 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:20.881830 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:20.881835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.882915 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.882918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:20.884096 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:20.884110 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.884790 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.884793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:20.884854 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:20.885339 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:20.885350 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:20.887229 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 14:43:20.887506 9311 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:20.881213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48023 (local address 127.0.98.62:44033)
0619 14:43:20.881352 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:20.881356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:20.881370 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:43:20.881409 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:20.881414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:20.881466 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:20.881556 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:20.881939 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.882778 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.884278 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:20.884613 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:20.884718 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:20.887372 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48023: BlockingRecv error: recv got EOF from 127.0.98.2:48023 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 14:43:20.887737 9442 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:43:20.982864 9445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:20.983131 9445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:20.983174 9445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:20.987236 9445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:20.987294 9445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:20.987390 9445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:43:20.993091 9445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42037
--tserver_master_addrs=127.0.98.62:44033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:20.994355 9445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:20.995386 9445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:21.005174 9451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:21.005205 9450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:21.005142 9453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:21.005496 9445 server_base.cc:1034] running on GCE node
I20260619 14:43:21.005849 9445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:21.006403 9445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:21.007514 9445 hybrid_clock.cc:648] HybridClock initialized: now 1781880201007472 us; error 52 us; skew 500 ppm
Jun 19 14:43:21 dist-test-slave-cr22 krb5kdc[8578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880201, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:21.010326 9445 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 14:43:21.011207 9445 webserver.cc:460] Webserver started at http://127.0.98.3:42037/ using document root <none> and password file <none>
I20260619 14:43:21.011808 9445 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:21.011881 9445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:21.015625 9445 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260619 14:43:21.017742 9460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:21.019237 9445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:43:21.019377 9445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "8a51fff4d77f466a84a18dc371bbb534"
format_stamp: "Formatted at 2026-06-19 14:43:20 on dist-test-slave-cr22"
server_key: "bfb05eec4cff1a374466bb7ca17591a4"
server_key_iv: "ec80ba71f678dc2020090c64ef785934"
server_key_version: "encryptionkey@0"
I20260619 14:43:21.019987 9445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:21.030865 9445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:21.034237 9445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:21.034466 9445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:21.035110 9445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:21.036051 9445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:21.036124 9445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:21.036192 9445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:21.036252 9445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:21.046169 9445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37481
I20260619 14:43:21.046213 9573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37481 every 8 connection(s)
I20260619 14:43:21.047257 9445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:43:21.048578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 9445
Jun 19 14:43:21 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:21 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:21.059816 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:21.049759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.0.98.62:44033)
0619 14:43:21.050036 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:21.050039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:21.050051 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:43:21.050113 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:21.050115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:21.050170 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:21.050300 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:21.051177 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.052048 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.053124 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.053451 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.053567 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:21.055174 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:21.055181 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:21.055197 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:21.055227 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:21.057456 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:21.057945 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:21.057954 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:21.057963 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:21.058115 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:21.058423 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:21.058428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:21.058430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:21.058900 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:21.059114 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:21.059362 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 14:43:21.059648 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":98,"threads_started":1}
W20260619 14:43:21.060017 9311 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:21.049360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33617 (local address 127.0.98.62:44033)
0619 14:43:21.049521 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:21.049526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:21.050528 (+ 1002us) server_negotiation.cc:407] Connection header received
0619 14:43:21.052225 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:21.052229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:21.052280 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:21.052369 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:21.053602 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.054289 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.055723 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.056032 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.056145 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:21.059930 (+ 3785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33617: BlockingRecv error: recv got EOF from 127.0.98.3:33617 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:43:21.060782 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:21.049868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.3:33617)
0619 14:43:21.050358 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:21.050413 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:21.051967 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:21.052499 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:21.052506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:21.052848 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:21.053436 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:21.053452 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.054418 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.054420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:21.055594 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:21.055607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.056183 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.056190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:21.056402 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:21.056959 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:21.056978 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:21.059805 (+ 2827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":99,"threads_started":1}
W20260619 14:43:21.061225 9574 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44033 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44033: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 14:43:21 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:21 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:21.067328 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:21.061674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.3:42573)
0619 14:43:21.061812 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:21.061844 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:21.061926 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:21.062154 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:21.062157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:21.062347 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:21.062456 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:21.062462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.063418 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.063421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:21.064200 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:21.064209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.064722 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.064726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:21.064805 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:21.065386 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:21.065398 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:21.067225 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 14:43:21.067519 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:21.061792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42573 (local address 127.0.98.62:44033)
0619 14:43:21.061914 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:21.061918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:21.061931 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:43:21.062006 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:21.062012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:21.062055 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:21.062141 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:21.062601 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.063290 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.064324 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.064534 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.064611 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:21.067353 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42573: BlockingRecv error: recv got EOF from 127.0.98.3:42573 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 14:43:21 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:21 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:21.072707 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:21.067921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.3:55297)
0619 14:43:21.068047 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:21.068061 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:21.068137 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:21.068444 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:21.068447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:21.068569 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:21.068635 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:21.068640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.069337 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.069340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:21.069918 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:21.069927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.070329 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.070332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:21.070387 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:21.070837 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:21.070848 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:21.072607 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 14:43:21.072932 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:21.068056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55297 (local address 127.0.98.62:44033)
0619 14:43:21.068224 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:21.068227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:21.068239 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:43:21.068268 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:21.068270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:21.068302 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:21.068371 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:21.068738 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.069217 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.070031 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.070235 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.070314 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:21.072822 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55297: BlockingRecv error: recv got EOF from 127.0.98.3:55297 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 14:43:21.072958 9574 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 14:43:21 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:21 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:21.708734 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:21.702498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:32967)
0619 14:43:21.702850 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:21.702864 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:21.702957 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:21.703275 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:21.703278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:21.703497 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:21.703598 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:21.703604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.704568 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.704572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:21.705175 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:21.705184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.705801 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.705804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:21.705873 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:21.706381 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:21.706393 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:21.708605 (+ 2212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
W20260619 14:43:21.709002 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:21.702640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32967 (local address 127.0.98.62:44033)
0619 14:43:21.702892 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:21.702896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:21.702912 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:43:21.703047 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:21.703049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:21.703093 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:21.703186 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:21.703776 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.704421 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.705349 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.705613 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.705779 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:21.708848 (+ 3069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:32967: BlockingRecv error: recv got EOF from 127.0.98.1:32967 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":108,"threads_started":1}
Jun 19 14:43:21 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:21 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:21.894197 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:21.888436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:43435)
0619 14:43:21.888688 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:21.888710 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:21.888814 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:21.889154 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:21.889160 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:21.889402 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:21.889600 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:21.889611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.890442 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.890449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:21.891105 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:21.891113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.891619 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.891623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:21.891720 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:21.892206 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:21.892217 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:21.894013 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":91,"threads_started":1}
W20260619 14:43:21.894238 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:21.888503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43435 (local address 127.0.98.62:44033)
0619 14:43:21.888638 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:21.888642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:21.888775 (+ 133us) server_negotiation.cc:407] Connection header received
0619 14:43:21.888921 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:21.888923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:21.888966 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:21.889037 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:21.889734 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.890328 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.891225 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:21.891502 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:21.891619 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:21.894134 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43435: BlockingRecv error: recv got EOF from 127.0.98.2:43435 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 14:43:22 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:22 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:22.079669 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:22.073647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.3:58383)
0619 14:43:22.074111 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:22.074131 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:22.074249 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:22.074516 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:22.074519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:22.074688 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:22.074847 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:22.074853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:22.075639 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:22.075642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:22.076295 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:22.076304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:22.076835 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:22.076839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:22.076940 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:22.077500 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:22.077511 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:22.079544 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":263,"threads_started":1}
W20260619 14:43:22.079829 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:22.073808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58383 (local address 127.0.98.62:44033)
0619 14:43:22.074006 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:22.074010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:22.074250 (+ 240us) server_negotiation.cc:407] Connection header received
0619 14:43:22.074326 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:22.074328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:22.074377 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:22.074455 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:22.074986 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:22.075530 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:22.076420 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:22.076694 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:22.076833 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:22.079681 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58383: BlockingRecv error: recv got EOF from 127.0.98.3:58383 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 19 14:43:22 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:22 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:22.715821 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:22.709602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:33399)
0619 14:43:22.709977 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:22.709991 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:22.710083 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:22.710335 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:22.710337 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:22.710519 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:22.710612 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:22.710617 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:22.711481 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:22.711484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:22.712123 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:22.712133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:22.712644 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:22.712646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:22.712762 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:22.713297 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:22.713310 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:22.715651 (+ 2341us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":96,"threads_started":1}
W20260619 14:43:22.715907 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:22.709674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33399 (local address 127.0.98.62:44033)
0619 14:43:22.710020 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:22.710024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:22.710037 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:43:22.710169 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:22.710171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:22.710213 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:22.710317 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:22.710763 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:22.711336 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:22.712277 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:22.712524 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:22.712628 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:22.715796 (+ 3168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33399: BlockingRecv error: recv got EOF from 127.0.98.1:33399 (error 108)
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":168,"threads_started":1}
Jun 19 14:43:22 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:22 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:22.900650 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:22.894962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:50155)
0619 14:43:22.895216 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:22.895231 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:22.895337 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:22.895585 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:22.895588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:22.895818 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:22.895915 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:22.895921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:22.896696 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:22.896702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:22.897285 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:22.897294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:22.897782 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:22.897785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:22.897856 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:22.898448 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:22.898470 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:22.900509 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":90,"threads_started":1}
W20260619 14:43:22.900776 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:22.895047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50155 (local address 127.0.98.62:44033)
0619 14:43:22.895181 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:22.895185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:22.895312 (+ 127us) server_negotiation.cc:407] Connection header received
0619 14:43:22.895415 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:22.895417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:22.895481 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:22.895575 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:22.896019 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:22.896524 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:22.897394 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:22.897611 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:22.897713 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:22.900645 (+ 2932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50155: BlockingRecv error: recv got EOF from 127.0.98.2:50155 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 14:43:23 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:23 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:23.092195 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:23.080578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.3:45765)
0619 14:43:23.081089 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:23.081111 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:23.081264 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:23.081560 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:23.081563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:23.081853 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:23.081948 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:23.081954 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:23.082759 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:23.082765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:23.083342 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:23.083350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:23.083847 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:23.083851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:23.083938 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:23.089533 (+ 5595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:23.089552 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:23.091955 (+ 2403us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":190,"threads_started":1}
W20260619 14:43:23.092305 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:23.080802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45765 (local address 127.0.98.62:44033)
0619 14:43:23.081059 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:23.081064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:23.081199 (+ 135us) server_negotiation.cc:407] Connection header received
0619 14:43:23.081375 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:23.081379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:23.081433 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:23.081533 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:23.082084 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:23.082638 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:23.083488 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:23.083719 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:23.083839 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:23.092172 (+ 8333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45765: BlockingRecv error: recv got EOF from 127.0.98.3:45765 (error 108)
Metrics: {"server-negotiator.queue_time_us":163}
Jun 19 14:43:23 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:23 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:23.723140 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:23.716873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:54137)
0619 14:43:23.717168 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:23.717182 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:23.717275 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:23.717523 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:23.717525 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:23.717706 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:23.717798 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:23.717803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:23.718648 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:23.718651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:23.719424 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:23.719501 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:23.720072 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:23.720076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:23.720171 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:23.720786 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:23.720799 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:23.722854 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":97,"threads_started":1}
W20260619 14:43:23.723269 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:23.717026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54137 (local address 127.0.98.62:44033)
0619 14:43:23.717286 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:23.717290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:23.717331 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:43:23.717370 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:23.717372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:23.717412 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:23.717484 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:23.717913 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:23.718499 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:23.719637 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:23.719932 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:23.720070 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:23.723028 (+ 2958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54137: BlockingRecv error: recv got EOF from 127.0.98.1:54137 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":90,"threads_started":1}
Jun 19 14:43:23 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:23 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:23.907686 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:23.901626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:42589)
0619 14:43:23.901932 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:23.901946 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:23.902053 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:23.902339 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:23.902341 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:23.902528 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:23.902627 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:23.902632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:23.903517 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:23.903520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:23.904089 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:23.904098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:23.904630 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:23.904633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:23.904700 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:23.905280 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:23.905300 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:23.907486 (+ 2186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":117,"threads_started":1}
W20260619 14:43:23.907828 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:23.901723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42589 (local address 127.0.98.62:44033)
0619 14:43:23.901898 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:23.901901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:23.902001 (+ 100us) server_negotiation.cc:407] Connection header received
0619 14:43:23.902153 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:23.902156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:23.902201 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:23.902272 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:23.902767 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:23.903306 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:23.904258 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:23.904502 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:23.904597 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:23.907641 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42589: BlockingRecv error: recv got EOF from 127.0.98.2:42589 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 14:43:24 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:24 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:24.099260 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:24.093276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.3:60507)
0619 14:43:24.093568 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:24.093582 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:24.093672 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:24.093983 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:24.093986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:24.094179 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:24.094282 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:24.094289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:24.095373 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:24.095377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:24.095995 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:24.096004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:24.096490 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:24.096493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:24.096570 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:24.097246 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:24.097258 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:24.099134 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":107,"threads_started":1}
W20260619 14:43:24.099498 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:24.093418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60507 (local address 127.0.98.62:44033)
0619 14:43:24.093653 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:24.093658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:24.093675 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:43:24.093761 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:24.093763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:24.093823 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:24.093950 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:24.094501 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:24.095219 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:24.096137 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:24.096375 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:24.096473 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:24.099309 (+ 2836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60507: BlockingRecv error: recv got EOF from 127.0.98.3:60507 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 14:43:24 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:24 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:24.730832 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:24.724118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:59597)
0619 14:43:24.724436 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:24.724452 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:24.724603 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:24.724993 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:24.724996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:24.725179 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:24.725276 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:24.725281 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:24.726407 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:24.726410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:24.727207 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:24.727221 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:24.727983 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:24.727986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:24.728054 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:24.728551 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:24.728563 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:24.730715 (+ 2152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":107,"threads_started":1}
W20260619 14:43:24.731123 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:24.724339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59597 (local address 127.0.98.62:44033)
0619 14:43:24.724640 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:24.724643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:24.724657 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:43:24.724792 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:24.724794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:24.724847 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:24.724935 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:24.725402 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:24.726279 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:24.727339 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:24.727824 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:24.727923 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:24.730980 (+ 3057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59597: BlockingRecv error: recv got EOF from 127.0.98.1:59597 (error 108)
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":139,"threads_started":1}
Jun 19 14:43:24 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:24 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:24.914351 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:24.908781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:54707)
0619 14:43:24.909047 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:24.909067 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:24.909195 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:24.909487 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:24.909490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:24.909701 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:24.909794 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:24.909799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:24.910551 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:24.910554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:24.911130 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:24.911137 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:24.911669 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:24.911672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:24.911757 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:24.912248 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:24.912260 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:24.914225 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":92,"threads_started":1}
W20260619 14:43:24.914476 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:24.908887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54707 (local address 127.0.98.62:44033)
0619 14:43:24.909013 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:24.909018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:24.909142 (+ 124us) server_negotiation.cc:407] Connection header received
0619 14:43:24.909310 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:24.909313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:24.909366 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:24.909457 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:24.909910 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:24.910439 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:24.911253 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:24.911545 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:24.911663 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:24.914361 (+ 2698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54707: BlockingRecv error: recv got EOF from 127.0.98.2:54707 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 14:43:25 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:25 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:25.106946 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:25.100227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.3:35289)
0619 14:43:25.100870 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:25.100884 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:25.100993 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:25.101294 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:25.101297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:25.101479 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:25.101577 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:25.101582 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:25.102374 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:25.102377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:25.102958 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:25.102967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:25.103516 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:25.103519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:25.103585 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:25.104196 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:25.104208 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:25.106816 (+ 2608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":113,"threads_started":1}
W20260619 14:43:25.107079 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:25.100324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35289 (local address 127.0.98.62:44033)
0619 14:43:25.100490 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:25.100495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:25.101013 (+ 518us) server_negotiation.cc:407] Connection header received
0619 14:43:25.101059 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:25.101062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:25.101112 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:25.101203 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:25.101709 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:25.102257 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:25.103086 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:25.103319 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:25.103407 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:25.106929 (+ 3522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35289: BlockingRecv error: recv got EOF from 127.0.98.3:35289 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 14:43:25 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:25 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:25.737691 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:25.731860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:51629)
0619 14:43:25.732250 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:25.732264 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:25.732359 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:25.732590 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:25.732592 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:25.732772 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:25.732875 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:25.732881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:25.733785 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:25.733788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:25.734363 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:25.734371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:25.734904 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:25.734907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:25.734974 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:25.735580 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:25.735592 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:25.737569 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":89,"threads_started":1}
W20260619 14:43:25.737913 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:25.731919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51629 (local address 127.0.98.62:44033)
0619 14:43:25.732213 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:25.732218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:25.732337 (+ 119us) server_negotiation.cc:407] Connection header received
0619 14:43:25.732446 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:25.732448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:25.732491 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:25.732559 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:25.733025 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:25.733666 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:25.734493 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:25.734759 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:25.734852 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:25.737690 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51629: BlockingRecv error: recv got EOF from 127.0.98.1:51629 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":65,"threads_started":1}
Jun 19 14:43:25 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:25 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:25.921258 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:25.915186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:34153)
0619 14:43:25.915505 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:25.915530 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:25.915635 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:25.915924 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:25.915926 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:25.916116 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:25.916214 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:25.916220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:25.917028 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:25.917031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:25.917723 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:25.917733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:25.918347 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:25.918350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:25.918439 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:25.919011 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:25.919028 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:25.921125 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":124,"threads_started":1}
W20260619 14:43:25.921525 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:25.915342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34153 (local address 127.0.98.62:44033)
0619 14:43:25.915529 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:25.915532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:25.915598 (+ 66us) server_negotiation.cc:407] Connection header received
0619 14:43:25.915730 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:25.915733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:25.915778 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:25.915850 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:25.916356 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:25.916906 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:25.917867 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:25.918153 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:25.918329 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:25.921288 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34153: BlockingRecv error: recv got EOF from 127.0.98.2:34153 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 14:43:26 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:26 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:26.113924 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:26.107958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.3:38249)
0619 14:43:26.108321 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:26.108335 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:26.108470 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:26.108779 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:26.108782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:26.109001 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:26.109104 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:26.109109 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:26.109962 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:26.109968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:26.110614 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:26.110623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:26.111123 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:26.111127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:26.111220 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:26.111785 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:26.111797 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:26.113717 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":94,"threads_started":1}
W20260619 14:43:26.113936 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:26.108056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38249 (local address 127.0.98.62:44033)
0619 14:43:26.108205 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:26.108211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:26.108419 (+ 208us) server_negotiation.cc:407] Connection header received
0619 14:43:26.108588 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:26.108592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:26.108650 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:26.108751 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:26.109232 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:26.109840 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:26.110757 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:26.111000 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:26.111122 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:26.113836 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38249: BlockingRecv error: recv got EOF from 127.0.98.3:38249 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 14:43:26 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:26 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:43:26.745671 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:26.738588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51763 (local address 127.0.98.62:44033)
0619 14:43:26.738822 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:26.738827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:26.738945 (+ 118us) server_negotiation.cc:407] Connection header received
0619 14:43:26.739158 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:26.739162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:26.739223 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:26.739313 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:26.739863 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:26.740487 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:26.742118 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:26.742349 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:26.742434 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:26.745559 (+ 3125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51763: BlockingRecv error: recv got EOF from 127.0.98.1:51763 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":53,"threads_started":1}
I20260619 14:43:26.745692 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:26.738468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:51763)
0619 14:43:26.738836 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:26.738853 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:26.739023 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:26.739335 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:26.739338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:26.739563 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:26.739740 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:26.739746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:26.740622 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:26.740629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:26.741935 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:26.741950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:26.742482 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:26.742488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:26.742615 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:26.743190 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:26.743211 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:26.745440 (+ 2229us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":90,"threads_started":1}
Jun 19 14:43:26 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:26 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:43:26.927956 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:26.922061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46211 (local address 127.0.98.62:44033)
0619 14:43:26.922248 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:26.922253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:26.922371 (+ 118us) server_negotiation.cc:407] Connection header received
0619 14:43:26.922555 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:26.922558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:26.922601 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:26.922684 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:26.923141 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:26.923900 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:26.924783 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:26.925021 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:26.925129 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:26.927864 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46211: BlockingRecv error: recv got EOF from 127.0.98.2:46211 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 14:43:26.927953 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:26.921934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:46211)
0619 14:43:26.922266 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:26.922283 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:26.922436 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:26.922703 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:26.922705 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:26.922922 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:26.923020 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:26.923026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:26.924019 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:26.924023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:26.924657 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:26.924666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:26.925144 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:26.925148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:26.925233 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:26.925880 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:26.925891 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:26.927795 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":82,"threads_started":1}
Jun 19 14:43:27 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:27 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:27.120530 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:27.114789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.3:39353)
0619 14:43:27.115170 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:27.115188 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:27.115362 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:27.115728 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:27.115734 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:27.115930 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:27.116020 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:27.116025 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:27.116789 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:27.116796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:27.117355 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:27.117363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:27.117815 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:27.117819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:27.117905 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:27.118433 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:27.118444 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:27.120414 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":87,"threads_started":1}
W20260619 14:43:27.120703 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:27.114925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39353 (local address 127.0.98.62:44033)
0619 14:43:27.115170 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:27.115176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:27.115292 (+ 116us) server_negotiation.cc:407] Connection header received
0619 14:43:27.115488 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:27.115493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:27.115549 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:27.115628 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:27.116139 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:27.116675 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:27.117476 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:27.117699 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:27.117812 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:27.120571 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39353: BlockingRecv error: recv got EOF from 127.0.98.3:39353 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Jun 19 14:43:27 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:27 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:27.752409 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:27.746416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:38897)
0619 14:43:27.746729 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:27.746749 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:27.746840 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:27.747095 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:27.747097 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:27.747272 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:27.747365 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:27.747370 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:27.748294 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:27.748297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:27.748929 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:27.748939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:27.749419 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:27.749422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:27.749486 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:27.750045 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:27.750057 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:27.752279 (+ 2222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":88,"threads_started":1}
W20260619 14:43:27.752589 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:27.746492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38897 (local address 127.0.98.62:44033)
0619 14:43:27.746831 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:27.746834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:27.746848 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:43:27.746931 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:27.746934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:27.746975 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:27.747059 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:27.747563 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:27.748171 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:27.749068 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:27.749297 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:27.749387 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:27.752443 (+ 3056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38897: BlockingRecv error: recv got EOF from 127.0.98.1:38897 (error 108)
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":191,"threads_started":1}
Jun 19 14:43:27 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:27 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:27.935628 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:27.928942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:48265)
0619 14:43:27.929393 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:27.929406 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:27.929508 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:27.929897 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:27.929900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:27.930094 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:27.930198 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:27.930205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:27.931124 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:27.931127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:27.931925 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:27.931937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:27.932515 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:27.932521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:27.932611 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:27.933247 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:27.933262 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:27.935503 (+ 2241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":158,"threads_started":1}
W20260619 14:43:27.935834 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:27.929105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48265 (local address 127.0.98.62:44033)
0619 14:43:27.929338 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:27.929341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:27.929489 (+ 148us) server_negotiation.cc:407] Connection header received
0619 14:43:27.929644 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:27.929647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:27.929701 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:27.929811 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:27.930394 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:27.930998 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:27.932060 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:27.932343 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:27.932471 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:27.935698 (+ 3227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48265: BlockingRecv error: recv got EOF from 127.0.98.2:48265 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 19 14:43:28 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:28 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:28.133039 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:28.121431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.3:36727)
0619 14:43:28.121755 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:28.121769 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:28.121892 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:28.122184 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:28.122186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:28.122363 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:28.122459 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:28.122464 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:28.123331 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:28.123357 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:28.124270 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:28.124281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:28.124901 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:28.124905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:28.125003 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:28.130697 (+ 5694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:28.130718 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:28.132912 (+ 2194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":70,"threads_started":1}
W20260619 14:43:28.133325 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:28.121575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36727 (local address 127.0.98.62:44033)
0619 14:43:28.121749 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:28.121753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:28.121879 (+ 126us) server_negotiation.cc:407] Connection header received
0619 14:43:28.122004 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:28.122007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:28.122059 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:28.122146 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:28.122574 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:28.123178 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:28.124429 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:28.124765 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:28.124918 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:28.133181 (+ 8263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36727: BlockingRecv error: recv got EOF from 127.0.98.3:36727 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 14:43:28 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:28 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:28.759068 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:28.753280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:44221)
0619 14:43:28.753572 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:28.753585 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:28.753675 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:28.753936 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:28.753939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:28.754122 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:28.754232 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:28.754238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:28.755032 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:28.755035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:28.755760 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:28.755770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:28.756249 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:28.756251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:28.756313 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:28.756912 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:28.756925 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:28.758947 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":115,"threads_started":1}
W20260619 14:43:28.759277 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:28.753422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44221 (local address 127.0.98.62:44033)
0619 14:43:28.753695 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:28.753699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:28.753713 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:43:28.753760 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:28.753762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:28.753804 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:28.753899 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:28.754354 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:28.754914 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:28.755911 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:28.756134 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:28.756231 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:28.759106 (+ 2875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44221: BlockingRecv error: recv got EOF from 127.0.98.1:44221 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":181,"threads_started":1}
Jun 19 14:43:28 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:28 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:28.943778 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:28.936596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:33097)
0619 14:43:28.937018 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:28.937033 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:28.937180 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:28.937494 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:28.937497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:28.937720 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:28.937830 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:28.937837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:28.939251 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:28.939258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:28.940160 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:28.940170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:28.940698 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:28.940702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:28.940794 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:28.941333 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:28.941346 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:28.943613 (+ 2267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":90,"threads_started":1}
W20260619 14:43:28.943908 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:28.936681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33097 (local address 127.0.98.62:44033)
0619 14:43:28.936956 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:28.936962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:28.937115 (+ 153us) server_negotiation.cc:407] Connection header received
0619 14:43:28.937306 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:28.937311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:28.937371 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:28.937471 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:28.937969 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:28.939114 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:28.940312 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:28.940571 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:28.940696 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:28.943796 (+ 3100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33097: BlockingRecv error: recv got EOF from 127.0.98.2:33097 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
Jun 19 14:43:29 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:29 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:29.139766 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:29.133820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.3:38417)
0619 14:43:29.134062 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:29.134076 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:29.134167 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:29.134422 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:29.134424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:29.134602 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:29.134694 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:29.134699 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:29.135493 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:29.135496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:29.136071 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:29.136080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:29.136561 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:29.136564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:29.136628 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:29.137307 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:29.137319 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:29.139644 (+ 2325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":63,"threads_started":1}
W20260619 14:43:29.139963 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:29.133951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38417 (local address 127.0.98.62:44033)
0619 14:43:29.134100 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:29.134103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:29.134121 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:43:29.134271 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:29.134273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:29.134316 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:29.134383 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:29.134815 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:29.135359 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:29.136204 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:29.136420 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:29.136512 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:29.139820 (+ 3308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38417: BlockingRecv error: recv got EOF from 127.0.98.3:38417 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 14:43:29 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:29 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:29.766482 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:29.759963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:50379)
0619 14:43:29.760297 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:29.760316 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:29.760435 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:29.760780 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:29.760783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:29.760972 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:29.761073 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:29.761078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:29.761920 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:29.761924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:29.762511 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:29.762522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:29.763098 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:29.763101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:29.763165 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:29.763846 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:29.763859 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:29.766347 (+ 2488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":99,"threads_started":1}
W20260619 14:43:29.766665 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:29.760173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50379 (local address 127.0.98.62:44033)
0619 14:43:29.760430 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:29.760434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:29.760448 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:43:29.760534 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:29.760536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:29.760579 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:29.760667 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:29.761205 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:29.761750 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:29.762667 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:29.762978 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:29.763066 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:29.766508 (+ 3442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50379: BlockingRecv error: recv got EOF from 127.0.98.1:50379 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
Jun 19 14:43:29 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:29 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:29.950959 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:29.944670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:37797)
0619 14:43:29.945020 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:29.945039 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:29.945170 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:29.945623 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:29.945626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:29.945829 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:29.945938 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:29.945944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:29.946734 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:29.946737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:29.947396 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:29.947405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:29.947973 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:29.947976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:29.948048 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:29.948566 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:29.948585 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:29.950820 (+ 2235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":116,"threads_started":1}
W20260619 14:43:29.951169 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:29.944891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37797 (local address 127.0.98.62:44033)
0619 14:43:29.945041 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:29.945047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:29.945101 (+ 54us) server_negotiation.cc:407] Connection header received
0619 14:43:29.945302 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:29.945305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:29.945376 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:29.945458 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:29.946060 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:29.946601 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:29.947572 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:29.947827 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:29.947922 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:29.950984 (+ 3062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37797: BlockingRecv error: recv got EOF from 127.0.98.2:37797 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 14:43:30 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:30 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:30.146493 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:30.140619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.3:35773)
0619 14:43:30.140943 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:30.140956 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:30.141085 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:30.141400 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:30.141403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:30.141617 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:30.141715 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:30.141720 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:30.142523 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:30.142528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:30.143096 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:30.143104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:30.143611 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:30.143615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:30.143694 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:30.144301 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:30.144313 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:30.146314 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":88,"threads_started":1}
W20260619 14:43:30.146539 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:30.140682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35773 (local address 127.0.98.62:44033)
0619 14:43:30.140856 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:30.140863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:30.141032 (+ 169us) server_negotiation.cc:407] Connection header received
0619 14:43:30.141192 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:30.141196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:30.141280 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:30.141382 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:30.141837 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:30.142410 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:30.143234 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:30.143496 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:30.143610 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:30.146438 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35773: BlockingRecv error: recv got EOF from 127.0.98.3:35773 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 14:43:30 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:30 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:30.773743 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:30.767527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:41323)
0619 14:43:30.767826 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:30.767846 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:30.767964 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:30.768246 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:30.768250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:30.768482 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:30.768758 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:30.768767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:30.769624 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:30.769627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:30.770212 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:30.770220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:30.770676 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:30.770679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:30.770747 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:30.771270 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:30.771283 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:30.773548 (+ 2265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":96,"threads_started":1}
W20260619 14:43:30.773954 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:30.767675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41323 (local address 127.0.98.62:44033)
0619 14:43:30.767925 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:30.767929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:30.767945 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:43:30.768055 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:30.768057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:30.768113 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:30.768192 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:30.768892 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:30.769453 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:30.770352 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:30.770572 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:30.770653 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:30.773781 (+ 3128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41323: BlockingRecv error: recv got EOF from 127.0.98.1:41323 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
Jun 19 14:43:30 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:30 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:30.957981 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:30.951904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:38967)
0619 14:43:30.952302 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:30.952318 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:30.952436 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:30.952792 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:30.952797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:30.953034 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:30.953170 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:30.953176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:30.953962 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:30.953965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:30.954568 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:30.954576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:30.955112 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:30.955115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:30.955185 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:30.955854 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:30.955866 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:30.957868 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":95,"threads_started":1}
W20260619 14:43:30.958134 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:30.951978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38967 (local address 127.0.98.62:44033)
0619 14:43:30.952169 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:30.952173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:30.952530 (+ 357us) server_negotiation.cc:407] Connection header received
0619 14:43:30.952572 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:30.952574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:30.952618 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:30.952731 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:30.953275 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:30.953844 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:30.954749 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:30.954967 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:30.955046 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:30.958014 (+ 2968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38967: BlockingRecv error: recv got EOF from 127.0.98.2:38967 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 14:43:31 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:31 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:31.153178 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:31.147345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.3:60357)
0619 14:43:31.147707 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:31.147725 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:31.147883 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:31.148173 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:31.148176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:31.148382 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:31.148472 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:31.148479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:31.149276 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:31.149281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:31.149851 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:31.149859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:31.150333 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:31.150336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:31.150422 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:31.150928 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:31.150940 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:31.153018 (+ 2078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":133,"threads_started":1}
W20260619 14:43:31.153345 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:31.147528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60357 (local address 127.0.98.62:44033)
0619 14:43:31.147717 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:31.147724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:31.147823 (+ 99us) server_negotiation.cc:407] Connection header received
0619 14:43:31.148005 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:31.148008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:31.148059 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:31.148144 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:31.148600 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:31.149164 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:31.149975 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:31.150213 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:31.150329 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:31.153192 (+ 2863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60357: BlockingRecv error: recv got EOF from 127.0.98.3:60357 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Jun 19 14:43:31 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:31 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:31.780560 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:31.774681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:51715)
0619 14:43:31.774978 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:31.774993 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:31.775100 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:31.775351 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:31.775353 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:31.775551 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:31.775647 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:31.775652 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:31.776414 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:31.776417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:31.777108 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:31.777119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:31.777577 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:31.777590 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:31.777732 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:31.778231 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:31.778243 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:31.780374 (+ 2131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
W20260619 14:43:31.780681 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:31.774697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51715 (local address 127.0.98.62:44033)
0619 14:43:31.774991 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:31.774995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:31.775071 (+ 76us) server_negotiation.cc:407] Connection header received
0619 14:43:31.775183 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:31.775185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:31.775229 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:31.775316 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:31.775758 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:31.776308 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:31.777230 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:31.777450 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:31.777533 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:31.780527 (+ 2994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51715: BlockingRecv error: recv got EOF from 127.0.98.1:51715 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":87,"threads_started":1}
Jun 19 14:43:31 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:31 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:31.964668 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:31.958766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:40321)
0619 14:43:31.959048 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:31.959063 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:31.959170 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:31.959390 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:31.959393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:31.959670 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:31.959770 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:31.959777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:31.960535 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:31.960539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:31.961186 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:31.961195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:31.961705 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:31.961708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:31.961778 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:31.962383 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:31.962395 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:31.964544 (+ 2149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":116,"threads_started":1}
W20260619 14:43:31.965000 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:31.958899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40321 (local address 127.0.98.62:44033)
0619 14:43:31.959046 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:31.959049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:31.959189 (+ 140us) server_negotiation.cc:407] Connection header received
0619 14:43:31.959230 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:31.959232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:31.959275 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:31.959348 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:31.959883 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:31.960405 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:31.961316 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:31.961532 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:31.961619 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:31.964798 (+ 3179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40321: BlockingRecv error: recv got EOF from 127.0.98.2:40321 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 14:43:32 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:32 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:32.159921 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:32.154014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.3:38639)
0619 14:43:32.154487 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:32.154500 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:32.154645 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:32.154985 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:32.154988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:32.155200 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:32.155298 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:32.155303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:32.156104 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:32.156110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:32.156678 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:32.156686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:32.157148 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:32.157152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:32.157236 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:32.157764 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:32.157776 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:32.159807 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":92,"threads_started":1}
W20260619 14:43:32.160084 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:32.154169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38639 (local address 127.0.98.62:44033)
0619 14:43:32.154435 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:32.154440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:32.154582 (+ 142us) server_negotiation.cc:407] Connection header received
0619 14:43:32.154752 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:32.154756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:32.154862 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:32.154964 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:32.155423 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:32.155987 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:32.156814 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:32.157038 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:32.157146 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:32.159954 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38639: BlockingRecv error: recv got EOF from 127.0.98.3:38639 (error 108)
Metrics: {"server-negotiator.queue_time_us":162}
Jun 19 14:43:32 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:32 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:32.787508 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:32.781663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:41331)
0619 14:43:32.781938 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:32.781952 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:32.782054 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:32.782276 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:32.782292 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:32.782470 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:32.782565 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:32.782571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:32.783511 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:32.783514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:32.784144 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:32.784154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:32.784635 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:32.784639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:32.784708 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:32.785246 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:32.785257 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:32.787331 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":97,"threads_started":1}
W20260619 14:43:32.787657 9623 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:32.781603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41331 (local address 127.0.98.62:44033)
0619 14:43:32.781884 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:32.781890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:32.782027 (+ 137us) server_negotiation.cc:407] Connection header received
0619 14:43:32.782115 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:32.782117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:32.782161 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:32.782239 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:32.782721 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:32.783322 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:32.784279 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:32.784505 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:32.784597 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:32.787516 (+ 2919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41331: BlockingRecv error: recv got EOF from 127.0.98.1:41331 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
Jun 19 14:43:32 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:32 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:32.971773 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:32.965545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:38981)
0619 14:43:32.965867 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:32.965880 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:32.965988 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:32.966412 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:32.966414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:32.966595 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:32.966691 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:32.966758 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:32.967569 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:32.967572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:32.968402 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:32.968413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:32.968863 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:32.968867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:32.968950 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:32.969502 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:32.969513 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:32.971604 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":99,"threads_started":1}
W20260619 14:43:32.971915 9623 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:32.965623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38981 (local address 127.0.98.62:44033)
0619 14:43:32.965779 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:32.965786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:32.965964 (+ 178us) server_negotiation.cc:407] Connection header received
0619 14:43:32.966226 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:32.966229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:32.966286 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:32.966362 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:32.966868 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:32.967394 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:32.968517 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:32.968747 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:32.968859 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:32.971799 (+ 2940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38981: BlockingRecv error: recv got EOF from 127.0.98.2:38981 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 14:43:33 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:33 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:33.170673 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:33.160998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.3:59119)
0619 14:43:33.161287 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:33.161301 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:33.161390 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:33.161787 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:33.161791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:33.162029 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:33.162132 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:33.162138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:33.162937 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:33.162940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:33.163590 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:33.163600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:33.164067 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:33.164070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:33.164133 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:33.168551 (+ 4418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:33.168564 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:33.170550 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":100,"threads_started":1}
W20260619 14:43:33.170917 9623 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:33.161214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59119 (local address 127.0.98.62:44033)
0619 14:43:33.161377 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:33.161382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:33.161395 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:43:33.161462 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:33.161465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:33.161519 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:33.161610 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:33.162240 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:33.162765 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:33.163710 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:33.163959 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:33.164043 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:33.170750 (+ 6707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59119: BlockingRecv error: recv got EOF from 127.0.98.3:59119 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 14:43:33 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:33 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:33.794486 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:33.788712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:47005)
0619 14:43:33.789008 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:33.789022 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:33.789159 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:33.789564 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:33.789572 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:33.789818 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:33.789965 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:33.789975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:33.790817 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:33.790820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:33.791381 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:33.791388 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:33.791874 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:33.791877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:33.791939 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:33.792427 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:33.792439 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:33.794312 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":106,"threads_started":1}
W20260619 14:43:33.794560 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:33.788842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47005 (local address 127.0.98.62:44033)
0619 14:43:33.789094 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:33.789098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:33.789112 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:43:33.789361 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:33.789364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:33.789423 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:33.789541 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:33.790093 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:33.790712 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:33.791557 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:33.791771 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:33.791850 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:33.794436 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47005: BlockingRecv error: recv got EOF from 127.0.98.1:47005 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":100,"threads_started":1}
Jun 19 14:43:33 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:33 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:33.978680 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:33.972747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:50227)
0619 14:43:33.973036 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:33.973052 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:33.973164 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:33.973460 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:33.973463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:33.973639 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:33.973749 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:33.973757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:33.974571 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:33.974574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:33.975143 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:33.975152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:33.975703 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:33.975707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:33.975788 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:33.976368 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:33.976381 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:33.978517 (+ 2136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":108,"threads_started":1}
W20260619 14:43:33.978883 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:33.972880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50227 (local address 127.0.98.62:44033)
0619 14:43:33.973053 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:33.973057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:33.973219 (+ 162us) server_negotiation.cc:407] Connection header received
0619 14:43:33.973285 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:33.973287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:33.973335 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:33.973413 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:33.973899 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:33.974434 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:33.975278 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:33.975585 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:33.975676 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:33.978680 (+ 3004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50227: BlockingRecv error: recv got EOF from 127.0.98.2:50227 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 19 14:43:34 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:34 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:34.177393 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:34.171573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.3:41311)
0619 14:43:34.171933 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:34.171957 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:34.172056 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:34.172351 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:34.172354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:34.172537 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:34.172694 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:34.172701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:34.173465 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:34.173467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:34.174089 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:34.174098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:34.174567 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:34.174570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:34.174717 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:34.175342 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:34.175363 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:34.177198 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":106,"threads_started":1}
W20260619 14:43:34.177457 9628 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:34.171644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41311 (local address 127.0.98.62:44033)
0619 14:43:34.171857 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:34.171862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:34.172031 (+ 169us) server_negotiation.cc:407] Connection header received
0619 14:43:34.172130 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:34.172132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:34.172181 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:34.172257 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:34.172836 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:34.173358 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:34.174217 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:34.174456 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:34.174540 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:34.177365 (+ 2825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41311: BlockingRecv error: recv got EOF from 127.0.98.3:41311 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 14:43:34 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:34 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:34.801126 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:34.795477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:33867)
0619 14:43:34.795774 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:34.795790 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:34.795891 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:34.796293 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:34.796295 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:34.796480 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:34.796594 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:34.796599 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:34.797416 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:34.797419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:34.798005 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:34.798013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:34.798451 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:34.798454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:34.798515 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:34.799032 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:34.799043 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:34.800935 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":79,"threads_started":1}
W20260619 14:43:34.801195 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:34.795752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33867 (local address 127.0.98.62:44033)
0619 14:43:34.796041 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:34.796045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:34.796057 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:43:34.796094 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:34.796096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:34.796136 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:34.796214 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:34.796753 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:34.797315 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:34.798110 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:34.798349 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:34.798426 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:34.801058 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33867: BlockingRecv error: recv got EOF from 127.0.98.1:33867 (error 108)
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":82,"threads_started":1}
Jun 19 14:43:34 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:34 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:34.985126 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:34.979540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:42817)
0619 14:43:34.979872 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:34.979892 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:34.980010 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:34.980231 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:34.980233 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:34.980411 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:34.980509 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:34.980515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:34.981262 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:34.981266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:34.981827 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:34.981835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:34.982328 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:34.982331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:34.982406 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:34.982906 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:34.982917 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:34.984991 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":89,"threads_started":1}
W20260619 14:43:34.985311 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:34.979627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42817 (local address 127.0.98.62:44033)
0619 14:43:34.979778 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:34.979785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:34.980011 (+ 226us) server_negotiation.cc:407] Connection header received
0619 14:43:34.980075 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:34.980077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:34.980123 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:34.980198 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:34.980620 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:34.981138 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:34.981960 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:34.982198 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:34.982290 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:34.985172 (+ 2882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42817: BlockingRecv error: recv got EOF from 127.0.98.2:42817 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 14:43:35 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:35 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:35.184401 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:35.178345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.3:58447)
0619 14:43:35.178643 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:35.178656 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:35.178753 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:35.179107 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:35.179111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:35.179290 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:35.179387 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:35.179392 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:35.180291 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:35.180294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:35.180901 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:35.180911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:35.181350 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:35.181353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:35.181419 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:35.182029 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:35.182041 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:35.184284 (+ 2243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":113,"threads_started":1}
W20260619 14:43:35.184536 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:35.178404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58447 (local address 127.0.98.62:44033)
0619 14:43:35.178570 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:35.178574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:35.178731 (+ 157us) server_negotiation.cc:407] Connection header received
0619 14:43:35.178880 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:35.178882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:35.178931 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:35.179007 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:35.179562 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:35.180175 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:35.181019 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:35.181238 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:35.181320 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:35.184413 (+ 3093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58447: BlockingRecv error: recv got EOF from 127.0.98.3:58447 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 14:43:35 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:35 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:35.807971 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:35.802111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:48551)
0619 14:43:35.802377 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:35.802390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:35.802469 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:35.802812 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:35.802815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:35.802995 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:35.803093 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:35.803099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:35.803918 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:35.803920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:35.804502 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:35.804511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:35.804982 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:35.804985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:35.805045 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:35.805535 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:35.805548 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:35.807856 (+ 2308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":96,"threads_started":1}
W20260619 14:43:35.808210 9636 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:35.802267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48551 (local address 127.0.98.62:44033)
0619 14:43:35.802491 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:35.802494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:35.802506 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:43:35.802540 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:35.802542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:35.802580 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:35.802679 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:35.803207 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:35.803790 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:35.804649 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:35.804871 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:35.804954 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:35.807989 (+ 3035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48551: BlockingRecv error: recv got EOF from 127.0.98.1:48551 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
Jun 19 14:43:35 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:43:35 dist-test-slave-cr22 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:43:35.992054 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:35.986291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:50937)
0619 14:43:35.986545 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:35.986559 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:35.986645 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:35.986913 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:35.986915 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:35.987090 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:35.987186 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:35.987192 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:35.987975 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:35.987978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:35.988691 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:35.988701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:35.989157 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:35.989161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:35.989226 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:35.989779 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:35.989790 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:35.991867 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44033: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":83,"threads_started":1}
W20260619 14:43:35.992094 9636 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:35.986379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50937 (local address 127.0.98.62:44033)
0619 14:43:35.986559 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:35.986563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:35.986609 (+ 46us) server_negotiation.cc:407] Connection header received
0619 14:43:35.986721 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:35.986723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:35.986763 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:35.986871 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:35.987289 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:35.987871 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:35.988833 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:35.989042 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:35.989122 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:35.991988 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50937: BlockingRecv error: recv got EOF from 127.0.98.2:50937 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:43:36.056394 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 9161
I20260619 14:43:36.062844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 9313
I20260619 14:43:36.068392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 9445
I20260619 14:43:36.074033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 9091
2026-06-19T14:43:36Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18092 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T14:43:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:43:36Z Disabled control of system clock
I20260619 14:43:36.105801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44331
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 14:43:36.218282 9641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:36.218544 9641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:36.218580 9641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:36.222779 9641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:43:36.222848 9641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:36.222865 9641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:36.222882 9641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:43:36.222899 9641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:43:36.227948 9641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:36.229130 9641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:36.230545 9641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:36.240065 9649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:36.240065 9646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:36.240213 9647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:36.240250 9641 server_base.cc:1034] running on GCE node
I20260619 14:43:36.241073 9641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:36.242249 9641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:36.243498 9641 hybrid_clock.cc:648] HybridClock initialized: now 1781880216243423 us; error 33 us; skew 500 ppm
I20260619 14:43:36.245514 9641 webserver.cc:460] Webserver started at http://127.0.98.62:41903/ using document root <none> and password file <none>
I20260619 14:43:36.246120 9641 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:36.246184 9641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:36.246407 9641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:36.248580 9641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "a400c0b9267849deb0c8e72b06e197c1"
format_stamp: "Formatted at 2026-06-19 14:43:36 on dist-test-slave-cr22"
server_key: "87ce16059d8767b8a6c99364d615f510"
server_key_iv: "bb2abd69b0cb46558c18558f0d5c8af2"
server_key_version: "encryptionkey@0"
I20260619 14:43:36.249168 9641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "a400c0b9267849deb0c8e72b06e197c1"
format_stamp: "Formatted at 2026-06-19 14:43:36 on dist-test-slave-cr22"
server_key: "87ce16059d8767b8a6c99364d615f510"
server_key_iv: "bb2abd69b0cb46558c18558f0d5c8af2"
server_key_version: "encryptionkey@0"
I20260619 14:43:36.253494 9641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 14:43:36.256770 9655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:36.258325 9641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:43:36.258515 9641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "a400c0b9267849deb0c8e72b06e197c1"
format_stamp: "Formatted at 2026-06-19 14:43:36 on dist-test-slave-cr22"
server_key: "87ce16059d8767b8a6c99364d615f510"
server_key_iv: "bb2abd69b0cb46558c18558f0d5c8af2"
server_key_version: "encryptionkey@0"
I20260619 14:43:36.258656 9641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:36.283965 9641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:36.284719 9641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:36.284943 9641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:36.292622 9641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44331
I20260619 14:43:36.292680 9707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44331 every 8 connection(s)
I20260619 14:43:36.293649 9641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:43:36.296905 9708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:36.302140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 9641
I20260619 14:43:36.302309 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:43:36.302631 392 external_mini_cluster.cc:1428] Setting key ade43c2fb7ad4d928ce3b94efc3fdf3a
I20260619 14:43:36.302842 9708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1: Bootstrap starting.
I20260619 14:43:36.305879 9708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:36.306913 9708 log.cc:824] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:36.309413 9708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1: No bootstrap required, opened a new log
I20260619 14:43:36.313869 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:36.304170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.0.98.62:44331)
0619 14:43:36.305094 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:36.305115 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:36.305164 (+ 49us) server_negotiation.cc:407] Connection header received
0619 14:43:36.306092 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:36.306115 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:36.306704 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:36.307128 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:36.308068 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:36.309598 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:36.310624 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:36.311109 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:36.311297 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:36.312178 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:36.312184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:36.312207 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:36.312605 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:36.312661 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:36.312681 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:36.312802 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:36.312910 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 14:43:36.313165 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":166,"threads_started":1}
I20260619 14:43:36.314019 9708 raft_consensus.cc:348] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a400c0b9267849deb0c8e72b06e197c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44331 } }
I20260619 14:43:36.314304 9708 raft_consensus.cc:374] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:36.314373 9708 raft_consensus.cc:729] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a400c0b9267849deb0c8e72b06e197c1, State: Initialized, Role: FOLLOWER
I20260619 14:43:36.314781 9708 consensus_queue.cc:260] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a400c0b9267849deb0c8e72b06e197c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44331 } }
I20260619 14:43:36.314929 9708 raft_consensus.cc:388] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:43:36.314993 9708 raft_consensus.cc:482] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:43:36.315090 9708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:36.316118 9708 raft_consensus.cc:504] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a400c0b9267849deb0c8e72b06e197c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44331 } }
I20260619 14:43:36.316457 9708 leader_election.cc:302] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a400c0b9267849deb0c8e72b06e197c1; no voters:
I20260619 14:43:36.316731 9708 leader_election.cc:288] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:43:36.316843 9713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:36.317135 9713 raft_consensus.cc:686] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1 [term 1 LEADER]: Becoming Leader. State: Replica: a400c0b9267849deb0c8e72b06e197c1, State: Running, Role: LEADER
I20260619 14:43:36.317500 9713 consensus_queue.cc:237] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a400c0b9267849deb0c8e72b06e197c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44331 } }
I20260619 14:43:36.317866 9708 sys_catalog.cc:564] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:43:36.319092 9715 sys_catalog.cc:455] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a400c0b9267849deb0c8e72b06e197c1. Latest consensus state: current_term: 1 leader_uuid: "a400c0b9267849deb0c8e72b06e197c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a400c0b9267849deb0c8e72b06e197c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44331 } } }
I20260619 14:43:36.319232 9715 sys_catalog.cc:458] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:36.319624 9714 sys_catalog.cc:455] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a400c0b9267849deb0c8e72b06e197c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a400c0b9267849deb0c8e72b06e197c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44331 } } }
I20260619 14:43:36.319733 9714 sys_catalog.cc:458] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1 [sys.catalog]: This master's current role is: LEADER
W20260619 14:43:36.322759 9728 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:43:36.322824 9728 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:43:36.322908 9724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:43:36.323688 9724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:43:36.328814 9724 catalog_manager.cc:1347] Generated new cluster ID: a835045c3d76437bbeec4bf137a55098
I20260619 14:43:36.328869 9724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:43:36.340320 9724 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:43:36.341317 9724 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:43:36.351478 9724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a400c0b9267849deb0c8e72b06e197c1: Generated new TSK 0
I20260619 14:43:36.352312 9724 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:43:36.364073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44331
--builtin_ntp_servers=127.0.98.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 14:43:36.477921 9732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:36.478223 9732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:36.478294 9732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:36.482790 9732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:36.482908 9732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:36.483016 9732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:43:36.488552 9732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:36.489850 9732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:36.491245 9732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:36.500801 9740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:36.500794 9737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:36.500900 9732 server_base.cc:1034] running on GCE node
W20260619 14:43:36.500794 9738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:36.501474 9732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:36.502136 9732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:36.503345 9732 hybrid_clock.cc:648] HybridClock initialized: now 1781880216503317 us; error 41 us; skew 500 ppm
I20260619 14:43:36.505334 9732 webserver.cc:460] Webserver started at http://127.0.98.1:39035/ using document root <none> and password file <none>
I20260619 14:43:36.505928 9732 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:36.506024 9732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:36.506269 9732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:36.508397 9732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "65988d7d87ec48cba9c03bf72fd36a58"
format_stamp: "Formatted at 2026-06-19 14:43:36 on dist-test-slave-cr22"
server_key: "3bc6354bba7109fa08c02fb6e5f5a1fb"
server_key_iv: "149998b7d2b03f4601bb67a98c883fee"
server_key_version: "encryptionkey@0"
I20260619 14:43:36.508967 9732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "65988d7d87ec48cba9c03bf72fd36a58"
format_stamp: "Formatted at 2026-06-19 14:43:36 on dist-test-slave-cr22"
server_key: "3bc6354bba7109fa08c02fb6e5f5a1fb"
server_key_iv: "149998b7d2b03f4601bb67a98c883fee"
server_key_version: "encryptionkey@0"
I20260619 14:43:36.512893 9732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:43:36.515555 9746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:36.517000 9732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:43:36.517141 9732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "65988d7d87ec48cba9c03bf72fd36a58"
format_stamp: "Formatted at 2026-06-19 14:43:36 on dist-test-slave-cr22"
server_key: "3bc6354bba7109fa08c02fb6e5f5a1fb"
server_key_iv: "149998b7d2b03f4601bb67a98c883fee"
server_key_version: "encryptionkey@0"
I20260619 14:43:36.517285 9732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:36.542649 9732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:36.543735 9732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:36.543946 9732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:36.544786 9732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:36.546355 9732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:36.546442 9732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:36.546517 9732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:36.546595 9732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 14:43:36.558720 9732 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44059
I20260619 14:43:36.558770 9859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44059 every 8 connection(s)
I20260619 14:43:36.559847 9732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:43:36.560693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 9732
I20260619 14:43:36.560832 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:43:36.561187 392 external_mini_cluster.cc:1428] Setting key 11ec1f61905b23d022ea059ccfdf8bd1
I20260619 14:43:36.563252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44331
--builtin_ntp_servers=127.0.98.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 14:43:36.571971 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:36.564728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51721 (local address 127.0.98.62:44331)
0619 14:43:36.564902 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:36.564909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:36.564928 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:43:36.566082 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:36.566088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:36.566149 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:36.566245 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:36.567268 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:36.568242 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:36.569611 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:36.570031 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:36.570238 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:36.571324 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:36.571329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:36.571332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:36.571349 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:36.571378 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:36.571390 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:36.571514 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:36.571687 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 14:43:36.571765 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 14:43:36.572824 9862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:36.562377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44331 (local address 127.0.98.1:51721)
0619 14:43:36.563240 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:36.563296 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:36.565861 (+ 2565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:36.566391 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:36.566398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:36.566477 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:36.567114 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:36.567128 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:36.568392 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:36.568395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:36.569425 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:36.569435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:36.570217 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:36.570222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:36.570387 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:36.571043 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:43:36.571045 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:36.571083 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:43:36.571149 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:36.571537 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:36.571544 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:36.571548 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:43:36.571648 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 14:43:36.571958 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":152,"threads_started":1}
I20260619 14:43:36.574154 9860 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44331
I20260619 14:43:36.574460 9860 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:36.574983 9860 heartbeater.cc:511] Master 127.0.98.62:44331 requested a full tablet report, sending...
I20260619 14:43:36.576647 9670 ts_manager.cc:194] Registered new tserver with Master: 65988d7d87ec48cba9c03bf72fd36a58 (127.0.98.1:44059)
I20260619 14:43:36.578208 9670 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51721
W20260619 14:43:36.678599 9863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:36.678881 9863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:36.678920 9863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:36.683102 9863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:36.683171 9863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:36.683233 9863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:43:36.688046 9863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:36.689072 9863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:36.690495 9863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:36.700449 9869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:36.700433 9868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:36.700426 9871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:36.701074 9863 server_base.cc:1034] running on GCE node
I20260619 14:43:36.701481 9863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:36.702147 9863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:36.703346 9863 hybrid_clock.cc:648] HybridClock initialized: now 1781880216703316 us; error 53 us; skew 500 ppm
I20260619 14:43:36.705411 9863 webserver.cc:460] Webserver started at http://127.0.98.2:34959/ using document root <none> and password file <none>
I20260619 14:43:36.706069 9863 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:36.706174 9863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:36.706403 9863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:36.708556 9863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "18a1b6e90f4c4caab8ba681ca0928f1d"
format_stamp: "Formatted at 2026-06-19 14:43:36 on dist-test-slave-cr22"
server_key: "9e4141e65c9b86e0e45f5539e89281ee"
server_key_iv: "b67dadd8e32d54523139a7822763b6b7"
server_key_version: "encryptionkey@0"
I20260619 14:43:36.709118 9863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "18a1b6e90f4c4caab8ba681ca0928f1d"
format_stamp: "Formatted at 2026-06-19 14:43:36 on dist-test-slave-cr22"
server_key: "9e4141e65c9b86e0e45f5539e89281ee"
server_key_iv: "b67dadd8e32d54523139a7822763b6b7"
server_key_version: "encryptionkey@0"
I20260619 14:43:36.712922 9863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:43:36.715780 9877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:36.717204 9863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:43:36.717478 9863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "18a1b6e90f4c4caab8ba681ca0928f1d"
format_stamp: "Formatted at 2026-06-19 14:43:36 on dist-test-slave-cr22"
server_key: "9e4141e65c9b86e0e45f5539e89281ee"
server_key_iv: "b67dadd8e32d54523139a7822763b6b7"
server_key_version: "encryptionkey@0"
I20260619 14:43:36.717607 9863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:36.732206 9863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:36.732940 9863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:36.733155 9863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:36.733793 9863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:36.734792 9863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:36.734864 9863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:36.734959 9863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:36.735011 9863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:36.744158 9863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43323
I20260619 14:43:36.744171 9990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43323 every 8 connection(s)
I20260619 14:43:36.745081 9863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:43:36.750029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 9863
I20260619 14:43:36.750190 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:43:36.750504 392 external_mini_cluster.cc:1428] Setting key b46b6bcc76b1accace757f13c2b8abc4
I20260619 14:43:36.752743 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44331
--builtin_ntp_servers=127.0.98.20:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 14:43:36.755811 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:36.746938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41313 (local address 127.0.98.62:44331)
0619 14:43:36.747073 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:36.747077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:36.748231 (+ 1154us) server_negotiation.cc:407] Connection header received
0619 14:43:36.749612 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:36.749615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:36.749673 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:36.749783 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:36.750793 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:36.751524 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:36.753411 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:36.753628 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:36.753729 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:36.755112 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:36.755118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:36.755121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:36.755138 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:36.755165 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:36.755177 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:36.755276 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:36.755557 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 14:43:36.755632 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:43:36.756630 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:36.747344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44331 (local address 127.0.98.2:41313)
0619 14:43:36.748066 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:36.748106 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:36.749398 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:36.749953 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:36.749960 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:36.750041 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:36.750634 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:36.750659 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:36.751683 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:36.751686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:36.753087 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:36.753102 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:36.753793 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:36.753800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:36.754037 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:36.754812 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:43:36.754814 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:36.754857 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:43:36.754926 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:36.755330 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:36.755338 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:36.755343 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:43:36.755538 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 14:43:36.755779 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":115,"threads_started":1}
I20260619 14:43:36.757988 9991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44331
I20260619 14:43:36.758288 9991 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:36.758841 9991 heartbeater.cc:511] Master 127.0.98.62:44331 requested a full tablet report, sending...
I20260619 14:43:36.760026 9670 ts_manager.cc:194] Registered new tserver with Master: 18a1b6e90f4c4caab8ba681ca0928f1d (127.0.98.2:43323)
I20260619 14:43:36.760937 9670 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41313
W20260619 14:43:36.868450 9994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:36.868721 9994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:36.868762 9994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:36.872797 9994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:36.872862 9994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:36.872916 9994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:43:36.877699 9994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:36.878939 9994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:36.880378 9994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:36.890136 9999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:36.890127 10000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:36.890127 10002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:36.890473 9994 server_base.cc:1034] running on GCE node
I20260619 14:43:36.890933 9994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:36.891660 9994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:36.892844 9994 hybrid_clock.cc:648] HybridClock initialized: now 1781880216892830 us; error 31 us; skew 500 ppm
I20260619 14:43:36.894757 9994 webserver.cc:460] Webserver started at http://127.0.98.3:34643/ using document root <none> and password file <none>
I20260619 14:43:36.895339 9994 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:36.895401 9994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:36.895659 9994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:36.897791 9994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "fb03c10d370344179e26738df79adc20"
format_stamp: "Formatted at 2026-06-19 14:43:36 on dist-test-slave-cr22"
server_key: "e3090dcc24e5e3d03f243bb47082f9d6"
server_key_iv: "66d5d5290629f0b433b36c667f39e658"
server_key_version: "encryptionkey@0"
I20260619 14:43:36.898350 9994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fb03c10d370344179e26738df79adc20"
format_stamp: "Formatted at 2026-06-19 14:43:36 on dist-test-slave-cr22"
server_key: "e3090dcc24e5e3d03f243bb47082f9d6"
server_key_iv: "66d5d5290629f0b433b36c667f39e658"
server_key_version: "encryptionkey@0"
I20260619 14:43:36.902005 9994 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 14:43:36.904505 10008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:36.905684 9994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 14:43:36.905829 9994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "fb03c10d370344179e26738df79adc20"
format_stamp: "Formatted at 2026-06-19 14:43:36 on dist-test-slave-cr22"
server_key: "e3090dcc24e5e3d03f243bb47082f9d6"
server_key_iv: "66d5d5290629f0b433b36c667f39e658"
server_key_version: "encryptionkey@0"
I20260619 14:43:36.905956 9994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:36.925149 9994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:36.925866 9994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:36.926069 9994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:36.926658 9994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:36.927661 9994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:36.927731 9994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:36.927802 9994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:36.927853 9994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:36.937393 9994 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38981
I20260619 14:43:36.937418 10121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38981 every 8 connection(s)
I20260619 14:43:36.938357 9994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:43:36.940027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 9994
I20260619 14:43:36.940171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:43:36.940470 392 external_mini_cluster.cc:1428] Setting key c92327e60ecfc9fa150e119e5aa8d3fc
I20260619 14:43:36.947944 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:36.940365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37785 (local address 127.0.98.62:44331)
0619 14:43:36.940522 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:36.940526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:36.941689 (+ 1163us) server_negotiation.cc:407] Connection header received
0619 14:43:36.943263 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:36.943266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:36.943318 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:36.943392 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:36.944282 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:36.944822 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:36.945950 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:36.946187 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:36.946298 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:36.947422 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:36.947425 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:36.947428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:36.947466 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:36.947487 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:36.947496 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:36.947584 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:36.947746 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 14:43:36.947805 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:43:36.948766 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:36.940811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44331 (local address 127.0.98.3:37785)
0619 14:43:36.941551 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:36.941600 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:36.943052 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:36.943577 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:36.943584 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:36.943665 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:36.944135 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:36.944147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:36.944949 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:36.944953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:36.945825 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:36.945834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:36.946335 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:36.946339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:36.946502 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:36.947140 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:43:36.947142 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:36.947180 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:43:36.947260 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:36.947586 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:36.947597 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:36.947607 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 14:43:36.947746 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 14:43:36.948017 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":150,"threads_started":1}
I20260619 14:43:36.949899 10122 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44331
I20260619 14:43:36.950163 10122 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:36.950704 10122 heartbeater.cc:511] Master 127.0.98.62:44331 requested a full tablet report, sending...
I20260619 14:43:36.951902 9670 ts_manager.cc:194] Registered new tserver with Master: fb03c10d370344179e26738df79adc20 (127.0.98.3:38981)
I20260619 14:43:36.952482 9670 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37785
I20260619 14:43:36.954627 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:43:36.957592 9711 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:36.956594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54346 (local address 127.0.98.62:44331)
0619 14:43:36.956726 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:36.956732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:36.956918 (+ 186us) server_negotiation.cc:407] Connection header received
0619 14:43:36.957079 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:36.957083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:36.957128 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:36.957213 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:36.957479 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54346: BlockingRecv error: recv got EOF from 127.0.0.1:54346 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:43:36.957809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 9732
I20260619 14:43:36.964183 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 9863
I20260619 14:43:36.969393 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 9994
I20260619 14:43:36.974720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 9641
2026-06-19T14:43:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (898 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T14:43:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:43:36Z Disabled control of system clock
I20260619 14:43:37.004347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36385
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 14:43:37.116575 10133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:37.116873 10133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:37.116910 10133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:37.121152 10133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:43:37.121215 10133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:37.121232 10133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:37.121248 10133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:43:37.121263 10133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:43:37.126173 10133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36385
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:37.127257 10133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:37.128713 10133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:37.137696 10138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:37.137714 10139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:37.137849 10141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:37.138021 10133 server_base.cc:1034] running on GCE node
I20260619 14:43:37.138579 10133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:37.139775 10133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:37.140952 10133 hybrid_clock.cc:648] HybridClock initialized: now 1781880217140934 us; error 32 us; skew 500 ppm
I20260619 14:43:37.142910 10133 webserver.cc:460] Webserver started at http://127.0.98.62:35479/ using document root <none> and password file <none>
I20260619 14:43:37.143558 10133 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:37.143630 10133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:37.143939 10133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:37.146037 10133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "ccb6406d1f7942588275d3eeccad0b75"
format_stamp: "Formatted at 2026-06-19 14:43:37 on dist-test-slave-cr22"
server_key: "65b8af85c8406e3385396d9a8c2620ea"
server_key_iv: "f940424e0dfa9d3fcedd3335b3c9cae0"
server_key_version: "encryptionkey@0"
I20260619 14:43:37.146549 10133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "ccb6406d1f7942588275d3eeccad0b75"
format_stamp: "Formatted at 2026-06-19 14:43:37 on dist-test-slave-cr22"
server_key: "65b8af85c8406e3385396d9a8c2620ea"
server_key_iv: "f940424e0dfa9d3fcedd3335b3c9cae0"
server_key_version: "encryptionkey@0"
I20260619 14:43:37.150460 10133 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 14:43:37.153271 10147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:37.154744 10133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:43:37.154856 10133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "ccb6406d1f7942588275d3eeccad0b75"
format_stamp: "Formatted at 2026-06-19 14:43:37 on dist-test-slave-cr22"
server_key: "65b8af85c8406e3385396d9a8c2620ea"
server_key_iv: "f940424e0dfa9d3fcedd3335b3c9cae0"
server_key_version: "encryptionkey@0"
I20260619 14:43:37.154963 10133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:37.159551 10133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:37.160125 10133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:37.160269 10133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:37.167174 10133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36385
I20260619 14:43:37.167208 10199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36385 every 8 connection(s)
I20260619 14:43:37.168195 10133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:43:37.170207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 10133
I20260619 14:43:37.170323 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:43:37.170639 392 external_mini_cluster.cc:1428] Setting key 4f9285afe26a4419af1347b0a60c0ac0
I20260619 14:43:37.172238 10200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:37.176357 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:37.172103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35192 (local address 127.0.98.62:36385)
0619 14:43:37.172743 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:37.172759 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:37.172792 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:43:37.173428 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:37.173443 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:37.173813 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:37.174090 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:37.174937 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:37.174942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:37.174955 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:37.174966 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:37.175023 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:37.175039 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:37.175159 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:37.175282 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 14:43:37.175593 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":146,"threads_started":1}
I20260619 14:43:37.178411 10200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75: Bootstrap starting.
I20260619 14:43:37.180711 10200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:37.181473 10200 log.cc:824] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:37.183878 10200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75: No bootstrap required, opened a new log
I20260619 14:43:37.187546 10200 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccb6406d1f7942588275d3eeccad0b75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36385 } }
I20260619 14:43:37.187844 10200 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:37.187937 10200 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb6406d1f7942588275d3eeccad0b75, State: Initialized, Role: FOLLOWER
I20260619 14:43:37.188442 10200 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccb6406d1f7942588275d3eeccad0b75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36385 } }
I20260619 14:43:37.188598 10200 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:43:37.188701 10200 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:43:37.188827 10200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:37.189831 10200 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccb6406d1f7942588275d3eeccad0b75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36385 } }
I20260619 14:43:37.190193 10200 leader_election.cc:302] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccb6406d1f7942588275d3eeccad0b75; no voters:
I20260619 14:43:37.190546 10200 leader_election.cc:288] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:43:37.190727 10205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:37.191071 10205 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75 [term 1 LEADER]: Becoming Leader. State: Replica: ccb6406d1f7942588275d3eeccad0b75, State: Running, Role: LEADER
I20260619 14:43:37.191382 10205 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccb6406d1f7942588275d3eeccad0b75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36385 } }
I20260619 14:43:37.192034 10200 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:43:37.193096 10207 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccb6406d1f7942588275d3eeccad0b75. Latest consensus state: current_term: 1 leader_uuid: "ccb6406d1f7942588275d3eeccad0b75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccb6406d1f7942588275d3eeccad0b75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36385 } } }
I20260619 14:43:37.193235 10207 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:37.193487 10206 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccb6406d1f7942588275d3eeccad0b75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccb6406d1f7942588275d3eeccad0b75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36385 } } }
I20260619 14:43:37.193595 10206 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:37.194612 10214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:43:37.197540 10214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:43:37.204080 10214 catalog_manager.cc:1347] Generated new cluster ID: 87e8af31c9b04a4f9f21011873601b5c
I20260619 14:43:37.204186 10214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:43:37.221418 10214 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:43:37.221786 10214 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:43:37.237488 10214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccb6406d1f7942588275d3eeccad0b75: Generated new TSK 0
I20260619 14:43:37.238310 10214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:43:37.249821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36385
--builtin_ntp_servers=127.0.98.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 14:43:37.364506 10224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:37.364777 10224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:37.364836 10224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:37.368928 10224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:37.368979 10224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:37.369035 10224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:43:37.373790 10224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:37.374861 10224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:37.376279 10224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:37.385731 10230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:37.385740 10229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:37.386006 10224 server_base.cc:1034] running on GCE node
W20260619 14:43:37.385740 10232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:37.386523 10224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:37.387100 10224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:37.388324 10224 hybrid_clock.cc:648] HybridClock initialized: now 1781880217388289 us; error 47 us; skew 500 ppm
I20260619 14:43:37.390213 10224 webserver.cc:460] Webserver started at http://127.0.98.1:35281/ using document root <none> and password file <none>
I20260619 14:43:37.390797 10224 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:37.390892 10224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:37.391119 10224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:37.393558 10224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "4b24a73502544ad8b52c4b79b25f05ff"
format_stamp: "Formatted at 2026-06-19 14:43:37 on dist-test-slave-cr22"
server_key: "73cce68bb400462182707e409a43f8ff"
server_key_iv: "005703f6b866ced72ade44b2b753c1fb"
server_key_version: "encryptionkey@0"
I20260619 14:43:37.394135 10224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4b24a73502544ad8b52c4b79b25f05ff"
format_stamp: "Formatted at 2026-06-19 14:43:37 on dist-test-slave-cr22"
server_key: "73cce68bb400462182707e409a43f8ff"
server_key_iv: "005703f6b866ced72ade44b2b753c1fb"
server_key_version: "encryptionkey@0"
I20260619 14:43:37.397859 10224 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 14:43:37.400391 10238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:37.401696 10224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 14:43:37.401842 10224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "4b24a73502544ad8b52c4b79b25f05ff"
format_stamp: "Formatted at 2026-06-19 14:43:37 on dist-test-slave-cr22"
server_key: "73cce68bb400462182707e409a43f8ff"
server_key_iv: "005703f6b866ced72ade44b2b753c1fb"
server_key_version: "encryptionkey@0"
I20260619 14:43:37.401962 10224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:37.406654 10224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:37.407240 10224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:37.407423 10224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:37.408017 10224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:37.409219 10224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:37.409291 10224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:37.409358 10224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:37.409390 10224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:37.417997 10224 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44043
I20260619 14:43:37.418017 10351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44043 every 8 connection(s)
I20260619 14:43:37.418879 10224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:43:37.425046 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:37.420889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45097 (local address 127.0.98.62:36385)
0619 14:43:37.421026 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:37.421029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:37.422032 (+ 1003us) server_negotiation.cc:407] Connection header received
0619 14:43:37.423313 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:37.423317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:37.423362 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:37.423467 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:37.424536 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:37.424540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:37.424542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:37.424562 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:37.424584 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:37.424592 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:37.424690 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:37.424867 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 14:43:37.424913 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:43:37.425853 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:37.421230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36385 (local address 127.0.98.1:45097)
0619 14:43:37.421889 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:37.421928 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:37.423091 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:37.423624 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:37.423631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:37.423721 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:37.424285 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:43:37.424286 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:37.424325 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:43:37.424390 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:37.424724 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:37.424732 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:37.424736 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:43:37.424898 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 14:43:37.425119 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20260619 14:43:37.426384 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 10224
I20260619 14:43:37.426512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:43:37.426841 392 external_mini_cluster.cc:1428] Setting key 59e6cca19e2a6c0ba85a546ab069d2d5
I20260619 14:43:37.427265 10352 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36385
I20260619 14:43:37.427527 10352 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:37.427930 10352 heartbeater.cc:511] Master 127.0.98.62:36385 requested a full tablet report, sending...
I20260619 14:43:37.428895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36385
--builtin_ntp_servers=127.0.98.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 14:43:37.429636 10164 ts_manager.cc:194] Registered new tserver with Master: 4b24a73502544ad8b52c4b79b25f05ff (127.0.98.1:44043)
W20260619 14:43:37.542083 10355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:37.542389 10355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:37.542431 10355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:37.546799 10355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:37.546865 10355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:37.546924 10355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:43:37.551714 10355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:37.552793 10355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:37.554118 10355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:37.564435 10360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:37.564350 10363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:37.564350 10361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:37.564508 10355 server_base.cc:1034] running on GCE node
I20260619 14:43:37.564991 10355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:37.565634 10355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:37.566836 10355 hybrid_clock.cc:648] HybridClock initialized: now 1781880217566818 us; error 39 us; skew 500 ppm
I20260619 14:43:37.568997 10355 webserver.cc:460] Webserver started at http://127.0.98.2:33751/ using document root <none> and password file <none>
I20260619 14:43:37.569593 10355 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:37.569689 10355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:37.569934 10355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:37.572147 10355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "a05b558f5e9c4c509becc83006957aa7"
format_stamp: "Formatted at 2026-06-19 14:43:37 on dist-test-slave-cr22"
server_key: "c30989b48661f2085900271cad47e062"
server_key_iv: "f6093254eb04d20f4743b3c6b950970f"
server_key_version: "encryptionkey@0"
I20260619 14:43:37.572726 10355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a05b558f5e9c4c509becc83006957aa7"
format_stamp: "Formatted at 2026-06-19 14:43:37 on dist-test-slave-cr22"
server_key: "c30989b48661f2085900271cad47e062"
server_key_iv: "f6093254eb04d20f4743b3c6b950970f"
server_key_version: "encryptionkey@0"
I20260619 14:43:37.576546 10355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:43:37.579185 10369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:37.580646 10355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:43:37.580809 10355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "a05b558f5e9c4c509becc83006957aa7"
format_stamp: "Formatted at 2026-06-19 14:43:37 on dist-test-slave-cr22"
server_key: "c30989b48661f2085900271cad47e062"
server_key_iv: "f6093254eb04d20f4743b3c6b950970f"
server_key_version: "encryptionkey@0"
I20260619 14:43:37.580940 10355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:37.586045 10355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:37.586697 10355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:37.586902 10355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:37.587575 10355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:37.588483 10355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:37.588557 10355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:37.588624 10355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:37.588685 10355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:37.597914 10355 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36093
I20260619 14:43:37.597937 10482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36093 every 8 connection(s)
I20260619 14:43:37.598807 10355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:43:37.605513 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:37.600928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34547 (local address 127.0.98.62:36385)
0619 14:43:37.601090 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:37.601094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:37.602246 (+ 1152us) server_negotiation.cc:407] Connection header received
0619 14:43:37.603559 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:37.603562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:37.603601 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:37.603672 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:37.604847 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:37.604854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:37.604859 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:37.604876 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:37.604923 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:37.604958 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:37.605064 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:37.605282 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 14:43:37.605326 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:43:37.605732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 10355
I20260619 14:43:37.605868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:43:37.606134 392 external_mini_cluster.cc:1428] Setting key e923a39eac4bd822732a0d36876dca48
I20260619 14:43:37.606563 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:37.601388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36385 (local address 127.0.98.2:34547)
0619 14:43:37.602062 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:37.602107 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:37.603290 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:37.603829 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:37.603835 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:37.603924 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:37.604476 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:43:37.604477 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:37.604517 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:43:37.604584 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:37.605093 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:37.605101 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:37.605106 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:43:37.605257 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 14:43:37.605521 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":122,"threads_started":1}
I20260619 14:43:37.607910 10483 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36385
I20260619 14:43:37.608168 10483 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:37.608374 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36385
--builtin_ntp_servers=127.0.98.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 14:43:37.608632 10483 heartbeater.cc:511] Master 127.0.98.62:36385 requested a full tablet report, sending...
I20260619 14:43:37.609735 10163 ts_manager.cc:194] Registered new tserver with Master: a05b558f5e9c4c509becc83006957aa7 (127.0.98.2:36093)
W20260619 14:43:37.719125 10486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:37.719381 10486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:37.719420 10486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:37.723529 10486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:37.723591 10486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:37.723647 10486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:43:37.728483 10486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:37.729568 10486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:37.730948 10486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:37.740698 10494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:37.740698 10491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:37.740734 10492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:37.741074 10486 server_base.cc:1034] running on GCE node
I20260619 14:43:37.741565 10486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:37.742231 10486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:37.743404 10486 hybrid_clock.cc:648] HybridClock initialized: now 1781880217743381 us; error 56 us; skew 500 ppm
I20260619 14:43:37.745467 10486 webserver.cc:460] Webserver started at http://127.0.98.3:46675/ using document root <none> and password file <none>
I20260619 14:43:37.746122 10486 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:37.746186 10486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:37.746443 10486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:37.748596 10486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "d5c805dd68b44cf4aaeb88780de21ecd"
format_stamp: "Formatted at 2026-06-19 14:43:37 on dist-test-slave-cr22"
server_key: "f53aa593e7179e92712e7bd6acb0d212"
server_key_iv: "5d8628b840f8361d9ede7190015775dc"
server_key_version: "encryptionkey@0"
I20260619 14:43:37.749176 10486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d5c805dd68b44cf4aaeb88780de21ecd"
format_stamp: "Formatted at 2026-06-19 14:43:37 on dist-test-slave-cr22"
server_key: "f53aa593e7179e92712e7bd6acb0d212"
server_key_iv: "5d8628b840f8361d9ede7190015775dc"
server_key_version: "encryptionkey@0"
I20260619 14:43:37.752944 10486 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:43:37.755637 10500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:37.757027 10486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:43:37.757185 10486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "d5c805dd68b44cf4aaeb88780de21ecd"
format_stamp: "Formatted at 2026-06-19 14:43:37 on dist-test-slave-cr22"
server_key: "f53aa593e7179e92712e7bd6acb0d212"
server_key_iv: "5d8628b840f8361d9ede7190015775dc"
server_key_version: "encryptionkey@0"
I20260619 14:43:37.757314 10486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:37.762049 10486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:37.762672 10486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:37.762852 10486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:37.763383 10486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:37.764317 10486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:37.764389 10486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:37.764453 10486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:37.764503 10486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:37.773280 10486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34169
I20260619 14:43:37.773360 10613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34169 every 8 connection(s)
I20260619 14:43:37.774247 10486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:43:37.775053 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 10486
I20260619 14:43:37.775178 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:43:37.775555 392 external_mini_cluster.cc:1428] Setting key df108fb9cd3db4b85b0451fc869af838
I20260619 14:43:37.781117 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:37.776307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58897 (local address 127.0.98.62:36385)
0619 14:43:37.776453 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:37.776459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:37.777880 (+ 1421us) server_negotiation.cc:407] Connection header received
0619 14:43:37.779195 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:37.779198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:37.779232 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:37.779310 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:37.780513 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:37.780516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:37.780518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:43:37.780528 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:37.780548 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:43:37.780556 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:37.780631 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:37.780974 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 14:43:37.781014 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:43:37.781826 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:37.776845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36385 (local address 127.0.98.3:58897)
0619 14:43:37.777725 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:37.777782 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:37.779002 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:37.779520 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:37.779536 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:37.779638 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:37.780265 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:43:37.780266 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:37.780306 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:43:37.780370 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:37.780681 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:37.780687 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:37.780691 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:43:37.780825 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 14:43:37.781006 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":149,"threads_started":1}
I20260619 14:43:37.782970 10614 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36385
I20260619 14:43:37.783234 10614 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:37.783699 10614 heartbeater.cc:511] Master 127.0.98.62:36385 requested a full tablet report, sending...
I20260619 14:43:37.784822 10163 ts_manager.cc:194] Registered new tserver with Master: d5c805dd68b44cf4aaeb88780de21ecd (127.0.98.3:34169)
I20260619 14:43:37.790371 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:43:37.793450 10203 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:43:37.792614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35206 (local address 127.0.98.62:36385)
0619 14:43:37.792809 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:37.792812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:37.792849 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:43:37.792981 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:37.792983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:37.793014 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:37.793081 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:37.793326 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35206: BlockingRecv error: recv got EOF from 127.0.0.1:35206 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
I20260619 14:43:37.793702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 10224
I20260619 14:43:37.799669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 10355
I20260619 14:43:37.805191 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 10486
I20260619 14:43:37.810853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 10133
2026-06-19T14:43:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (836 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:43:37 dist-test-slave-cr22 krb5kdc[10624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:43:37 dist-test-slave-cr22 krb5kdc[10624](info): set up 2 sockets
Jun 19 14:43:37 dist-test-slave-cr22 krb5kdc[10624](info): commencing operation
krb5kdc: starting...
W20260619 14:43:38.876824 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:43:38 dist-test-slave-cr22 krb5kdc[10624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:43:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:43:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:43:39.002681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39685
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:39.114935 10637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:39.115265 10637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:39.115341 10637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:39.119411 10637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:43:39.119525 10637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:39.119565 10637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:39.119601 10637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:43:39.119634 10637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:43:39.125015 10637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:39.126117 10637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:39.127159 10637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:39.136217 10645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:39.136217 10642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:39.136219 10643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:39.136521 10637 server_base.cc:1034] running on GCE node
I20260619 14:43:39.137176 10637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:39.138276 10637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:39.139525 10637 hybrid_clock.cc:648] HybridClock initialized: now 1781880219139507 us; error 31 us; skew 500 ppm
Jun 19 14:43:39 dist-test-slave-cr22 krb5kdc[10624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:39.147333 10637 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:43:39.148348 10637 webserver.cc:460] Webserver started at http://127.0.98.62:45911/ using document root <none> and password file <none>
I20260619 14:43:39.148948 10637 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:39.149042 10637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:39.149315 10637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:39.151551 10637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "46be1f8b797e43b7a4988b3f3fbe99e0"
format_stamp: "Formatted at 2026-06-19 14:43:39 on dist-test-slave-cr22"
server_key: "fa5caaa144eaacf65c179d07c6f767ba"
server_key_iv: "6554e24b13b35a78ed7099dae1380e44"
server_key_version: "encryptionkey@0"
I20260619 14:43:39.152136 10637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "46be1f8b797e43b7a4988b3f3fbe99e0"
format_stamp: "Formatted at 2026-06-19 14:43:39 on dist-test-slave-cr22"
server_key: "fa5caaa144eaacf65c179d07c6f767ba"
server_key_iv: "6554e24b13b35a78ed7099dae1380e44"
server_key_version: "encryptionkey@0"
I20260619 14:43:39.156045 10637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 14:43:39.158564 10652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:39.160053 10637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:43:39.160199 10637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "46be1f8b797e43b7a4988b3f3fbe99e0"
format_stamp: "Formatted at 2026-06-19 14:43:39 on dist-test-slave-cr22"
server_key: "fa5caaa144eaacf65c179d07c6f767ba"
server_key_iv: "6554e24b13b35a78ed7099dae1380e44"
server_key_version: "encryptionkey@0"
I20260619 14:43:39.160316 10637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:39.174517 10637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:39.177858 10637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:39.178066 10637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:39.185359 10637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39685
I20260619 14:43:39.185714 10704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39685 every 8 connection(s)
I20260619 14:43:39.186329 10637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:43:39.189193 10705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:39.189409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 10637
I20260619 14:43:39.189527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:43:39.189810 392 external_mini_cluster.cc:1428] Setting key d076808b6ec086dc763db72decdd4d90
I20260619 14:43:39.195847 10705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0: Bootstrap starting.
I20260619 14:43:39.198720 10705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0: Neither blocks nor log segments found. Creating new log.
Jun 19 14:43:39 dist-test-slave-cr22 krb5kdc[10624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:43:39.199541 10705 log.cc:824] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:39.201910 10705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0: No bootstrap required, opened a new log
I20260619 14:43:39.205093 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:39.191145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46424 (local address 127.0.98.62:39685)
0619 14:43:39.191554 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:39.191571 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:39.191604 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:43:39.192287 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:39.192303 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:39.192634 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:39.192906 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:39.193834 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:39.194813 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:39.196077 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:39.196592 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:39.196739 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:39.199689 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:39.199696 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:39.199711 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:39.199751 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:39.201801 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:39.202311 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:39.202316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:39.202322 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:39.202461 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:39.202741 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:39.202744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:39.202746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:39.203237 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:39.203426 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:39.203729 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 14:43:39.204057 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":130,"threads_started":1}
I20260619 14:43:39.206514 10705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46be1f8b797e43b7a4988b3f3fbe99e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39685 } }
I20260619 14:43:39.206720 10705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:39.206789 10705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46be1f8b797e43b7a4988b3f3fbe99e0, State: Initialized, Role: FOLLOWER
I20260619 14:43:39.207299 10705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46be1f8b797e43b7a4988b3f3fbe99e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39685 } }
I20260619 14:43:39.207479 10705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:43:39.207549 10705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:43:39.207635 10705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:39.208527 10705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46be1f8b797e43b7a4988b3f3fbe99e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39685 } }
I20260619 14:43:39.208870 10705 leader_election.cc:302] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46be1f8b797e43b7a4988b3f3fbe99e0; no voters:
I20260619 14:43:39.209165 10705 leader_election.cc:288] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:43:39.209368 10710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:39.209615 10710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0 [term 1 LEADER]: Becoming Leader. State: Replica: 46be1f8b797e43b7a4988b3f3fbe99e0, State: Running, Role: LEADER
I20260619 14:43:39.209895 10710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46be1f8b797e43b7a4988b3f3fbe99e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39685 } }
I20260619 14:43:39.210376 10705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:43:39.211642 10712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46be1f8b797e43b7a4988b3f3fbe99e0. Latest consensus state: current_term: 1 leader_uuid: "46be1f8b797e43b7a4988b3f3fbe99e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46be1f8b797e43b7a4988b3f3fbe99e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39685 } } }
I20260619 14:43:39.211777 10712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0 [sys.catalog]: This master's current role is: LEADER
I20260619 14:43:39.212023 10711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46be1f8b797e43b7a4988b3f3fbe99e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46be1f8b797e43b7a4988b3f3fbe99e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39685 } } }
I20260619 14:43:39.212177 10711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0 [sys.catalog]: This master's current role is: LEADER
W20260619 14:43:39.215512 10725 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:43:39.215590 10725 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:43:39.215667 10719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:43:39.216362 10719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:43:39.221689 10719 catalog_manager.cc:1347] Generated new cluster ID: 5af26651e03c447e871f3df019046816
I20260619 14:43:39.221766 10719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:43:39.249320 10719 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:43:39.250422 10719 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:43:39.255656 10719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0: Generated new TSK 0
I20260619 14:43:39.256407 10719 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:43:39.291870 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39685
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:39.431049 10731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:39.431324 10731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:39.431370 10731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:39.435565 10731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:39.435627 10731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:39.435688 10731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:43:39.440996 10731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:39.442015 10731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:39.443024 10731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:39.452538 10737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:39.452669 10739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:39.452570 10736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:39.452934 10731 server_base.cc:1034] running on GCE node
I20260619 14:43:39.453352 10731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:39.454025 10731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:39.455232 10731 hybrid_clock.cc:648] HybridClock initialized: now 1781880219455198 us; error 46 us; skew 500 ppm
Jun 19 14:43:39 dist-test-slave-cr22 krb5kdc[10624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:39.458212 10731 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:43:39.459062 10731 webserver.cc:460] Webserver started at http://127.0.98.1:40611/ using document root <none> and password file <none>
I20260619 14:43:39.459697 10731 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:39.459785 10731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:39.460005 10731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:39.461971 10731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "ca198a1372434e2b82f2b384f9e916df"
format_stamp: "Formatted at 2026-06-19 14:43:39 on dist-test-slave-cr22"
server_key: "9492b0119903a748947d8a4bcb88ee60"
server_key_iv: "05aacea25429a7c95e3d09591f17b249"
server_key_version: "encryptionkey@0"
I20260619 14:43:39.462530 10731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ca198a1372434e2b82f2b384f9e916df"
format_stamp: "Formatted at 2026-06-19 14:43:39 on dist-test-slave-cr22"
server_key: "9492b0119903a748947d8a4bcb88ee60"
server_key_iv: "05aacea25429a7c95e3d09591f17b249"
server_key_version: "encryptionkey@0"
I20260619 14:43:39.466368 10731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:43:39.468873 10746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:39.470142 10731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:43:39.470295 10731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "ca198a1372434e2b82f2b384f9e916df"
format_stamp: "Formatted at 2026-06-19 14:43:39 on dist-test-slave-cr22"
server_key: "9492b0119903a748947d8a4bcb88ee60"
server_key_iv: "05aacea25429a7c95e3d09591f17b249"
server_key_version: "encryptionkey@0"
I20260619 14:43:39.470422 10731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:39.492067 10731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:39.495337 10731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:39.495592 10731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:39.496186 10731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:39.497021 10731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:39.497077 10731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:39.497119 10731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:39.497174 10731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:39.506137 10731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34311
I20260619 14:43:39.506179 10859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34311 every 8 connection(s)
I20260619 14:43:39.507035 10731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:43:39.510301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 10731
I20260619 14:43:39.510425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:43:39.510692 392 external_mini_cluster.cc:1428] Setting key beb89a3bb3298d62be57a061e1a2c44a
Jun 19 14:43:39 dist-test-slave-cr22 krb5kdc[10624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:43:39.521195 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:39.508867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35591 (local address 127.0.98.62:39685)
0619 14:43:39.508996 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:39.509000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:39.510058 (+ 1058us) server_negotiation.cc:407] Connection header received
0619 14:43:39.511200 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:39.511203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:39.511262 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:39.511374 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:39.512715 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:39.513248 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:39.514440 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:39.514669 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:39.514775 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:39.517170 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:39.517174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:39.517177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:39.517205 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:39.518884 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:39.519411 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:39.519415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:39.519417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:39.519533 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:39.520023 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:39.520026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:39.520028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:39.520224 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:39.520346 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:39.520881 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 14:43:39.521018 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:43:39.522185 10862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:39.509162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39685 (local address 127.0.98.1:35591)
0619 14:43:39.509831 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:39.509952 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:39.510949 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:39.511578 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:39.511584 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:39.512056 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:39.512567 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:39.512579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:39.513417 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:39.513422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:39.514305 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:39.514314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:39.514821 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:39.514825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:39.514996 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:39.515620 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:39.515635 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:39.517008 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:39.519010 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:39.519015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:39.519035 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:39.519294 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:39.519652 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:39.519662 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:39.519664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:39.519877 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:39.520344 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:39.520351 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:39.520623 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 14:43:39.520840 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 14:43:39.521173 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:43:39.523936 10860 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39685
I20260619 14:43:39.524217 10860 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:39.524953 10860 heartbeater.cc:511] Master 127.0.98.62:39685 requested a full tablet report, sending...
I20260619 14:43:39.526614 10669 ts_manager.cc:194] Registered new tserver with Master: ca198a1372434e2b82f2b384f9e916df (127.0.98.1:34311)
I20260619 14:43:39.527822 10669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:43:39.539888 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39685
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:39.654310 10865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:39.654594 10865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:39.654639 10865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:39.658988 10865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:39.659056 10865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:39.659117 10865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:43:39.664592 10865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:39.665686 10865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:39.666672 10865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:43:39.676791 10870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:39.676824 10873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:39.677053 10865 server_base.cc:1034] running on GCE node
W20260619 14:43:39.677160 10871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:39.677647 10865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:39.678334 10865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:39.679530 10865 hybrid_clock.cc:648] HybridClock initialized: now 1781880219679502 us; error 41 us; skew 500 ppm
Jun 19 14:43:39 dist-test-slave-cr22 krb5kdc[10624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:39.682564 10865 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:43:39.683634 10865 webserver.cc:460] Webserver started at http://127.0.98.2:34723/ using document root <none> and password file <none>
I20260619 14:43:39.684230 10865 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:39.684324 10865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:39.684566 10865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:39.686790 10865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "84a710f2d902404791f2a0fe5aa6b0a6"
format_stamp: "Formatted at 2026-06-19 14:43:39 on dist-test-slave-cr22"
server_key: "74b91066214b2fe452d4508e30cb47d1"
server_key_iv: "eea210487a597f730554bd83c4dc8be0"
server_key_version: "encryptionkey@0"
I20260619 14:43:39.687373 10865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "84a710f2d902404791f2a0fe5aa6b0a6"
format_stamp: "Formatted at 2026-06-19 14:43:39 on dist-test-slave-cr22"
server_key: "74b91066214b2fe452d4508e30cb47d1"
server_key_iv: "eea210487a597f730554bd83c4dc8be0"
server_key_version: "encryptionkey@0"
I20260619 14:43:39.691206 10865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:43:39.694135 10880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:39.696051 10865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:43:39.696229 10865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "84a710f2d902404791f2a0fe5aa6b0a6"
format_stamp: "Formatted at 2026-06-19 14:43:39 on dist-test-slave-cr22"
server_key: "74b91066214b2fe452d4508e30cb47d1"
server_key_iv: "eea210487a597f730554bd83c4dc8be0"
server_key_version: "encryptionkey@0"
I20260619 14:43:39.696364 10865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:39.713150 10865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:39.716338 10865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:39.716760 10865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:39.717487 10865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:39.718355 10865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:39.718430 10865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:39.718503 10865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:39.718546 10865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:39.727943 10865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45019
I20260619 14:43:39.727973 10993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45019 every 8 connection(s)
I20260619 14:43:39.728876 10865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:43:39.736824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 10865
I20260619 14:43:39.736975 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:43:39.737319 392 external_mini_cluster.cc:1428] Setting key 5e933a4c0b6105ce78fe7aa41ae16dfb
Jun 19 14:43:39 dist-test-slave-cr22 krb5kdc[10624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:43:39.744131 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:39.730908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37815 (local address 127.0.98.62:39685)
0619 14:43:39.731054 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:39.731059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:39.732179 (+ 1120us) server_negotiation.cc:407] Connection header received
0619 14:43:39.733063 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:39.733066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:39.733125 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:39.733275 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:39.734417 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:39.734960 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:39.736197 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:39.736429 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:39.736555 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:39.739720 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:39.739727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:39.739733 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:39.739766 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:39.741390 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:39.742253 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:39.742260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:39.742269 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:39.742367 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:39.742738 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:39.742744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:39.742749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:39.743016 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:39.743121 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:39.743797 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 14:43:39.743937 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:43:39.745064 10996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:39.731223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39685 (local address 127.0.98.2:37815)
0619 14:43:39.732020 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:39.732062 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:39.732861 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:39.733437 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:39.733444 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:39.733802 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:39.734264 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:39.734277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:39.735088 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:39.735094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:39.736061 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:39.736070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:39.736597 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:39.736601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:39.736787 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:39.737544 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:39.737560 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:39.739542 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:39.741522 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:39.741531 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:39.741552 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:39.742080 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:39.742471 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:39.742474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:39.742475 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:39.742620 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:39.743131 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:39.743137 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:39.743440 (+ 303us) client_negotiation.cc:769] Sending connection context
0619 14:43:39.743680 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 14:43:39.743984 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":181,"threads_started":1}
I20260619 14:43:39.746387 10994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39685
I20260619 14:43:39.746665 10994 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:39.747206 10994 heartbeater.cc:511] Master 127.0.98.62:39685 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:43:39.748363 10669 ts_manager.cc:194] Registered new tserver with Master: 84a710f2d902404791f2a0fe5aa6b0a6 (127.0.98.2:45019)
I20260619 14:43:39.749189 10669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:43:39.766878 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39685
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:43:39.881358 10999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:43:39.881631 10999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:43:39.881675 10999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:43:39.885747 10999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:43:39.885807 10999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:43:39.885864 10999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:43:39.890964 10999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:43:39.891987 10999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:43:39.892947 10999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:43:39.903321 10999 server_base.cc:1034] running on GCE node
W20260619 14:43:39.903272 11007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:39.903272 11005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:43:39.903291 11004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:43:39.904062 10999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:43:39.904754 10999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:43:39.905968 10999 hybrid_clock.cc:648] HybridClock initialized: now 1781880219905949 us; error 35 us; skew 500 ppm
Jun 19 14:43:39 dist-test-slave-cr22 krb5kdc[10624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:43:39.909191 10999 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:43:39.910418 10999 webserver.cc:460] Webserver started at http://127.0.98.3:37183/ using document root <none> and password file <none>
I20260619 14:43:39.911186 10999 fs_manager.cc:359] Metadata directory not provided
I20260619 14:43:39.911293 10999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:43:39.911597 10999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:43:39.913707 10999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "b43cb7c8e49049849e6066333ff5f011"
format_stamp: "Formatted at 2026-06-19 14:43:39 on dist-test-slave-cr22"
server_key: "38c6e43defc9138e42998118e2d2bc6c"
server_key_iv: "70c86ff5651cdb5189e6e517a79e5561"
server_key_version: "encryptionkey@0"
I20260619 14:43:39.914276 10999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b43cb7c8e49049849e6066333ff5f011"
format_stamp: "Formatted at 2026-06-19 14:43:39 on dist-test-slave-cr22"
server_key: "38c6e43defc9138e42998118e2d2bc6c"
server_key_iv: "70c86ff5651cdb5189e6e517a79e5561"
server_key_version: "encryptionkey@0"
I20260619 14:43:39.918185 10999 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 14:43:39.920903 11014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:39.922291 10999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:43:39.922444 10999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "b43cb7c8e49049849e6066333ff5f011"
format_stamp: "Formatted at 2026-06-19 14:43:39 on dist-test-slave-cr22"
server_key: "38c6e43defc9138e42998118e2d2bc6c"
server_key_iv: "70c86ff5651cdb5189e6e517a79e5561"
server_key_version: "encryptionkey@0"
I20260619 14:43:39.922574 10999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:43:39.954854 10999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:43:39.958144 10999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:43:39.958356 10999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:43:39.959002 10999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:43:39.960047 10999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:43:39.960116 10999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:39.960178 10999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:43:39.960229 10999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:43:39.970050 10999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36325
I20260619 14:43:39.970078 11127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36325 every 8 connection(s)
I20260619 14:43:39.970968 10999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:43:39.974195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 10999
I20260619 14:43:39.974336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:43:39.974614 392 external_mini_cluster.cc:1428] Setting key 12ecce17c5e339a468b3ab32c8f89646
Jun 19 14:43:39 dist-test-slave-cr22 krb5kdc[10624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:43:39.985212 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:39.972852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52013 (local address 127.0.98.62:39685)
0619 14:43:39.973006 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:39.973010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:39.974148 (+ 1138us) server_negotiation.cc:407] Connection header received
0619 14:43:39.975160 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:39.975163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:39.975217 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:39.975310 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:39.976958 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:39.977487 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:39.978685 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:39.978895 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:39.978979 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:39.981622 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:39.981625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:39.981627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:39.981653 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:39.983150 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:39.983681 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:39.983684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:39.983686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:39.983762 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:39.984119 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:39.984122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:39.984123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:39.984322 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:39.984399 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:39.984962 (+ 563us) server_negotiation.cc:299] Negotiation successful
0619 14:43:39.985074 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:43:39.986069 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:39.973220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39685 (local address 127.0.98.3:52013)
0619 14:43:39.973966 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:39.974021 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:39.974895 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:39.975554 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:39.975564 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:39.976108 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:43:39.976806 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:39.976826 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:39.977601 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:39.977604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:39.978567 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:39.978576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:39.979048 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:39.979051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:39.979199 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:39.979951 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:43:39.979967 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:43:39.981462 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:43:39.983262 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:39.983267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:39.983279 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:39.983594 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:39.983869 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:43:39.983872 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:43:39.983873 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:43:39.984015 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:43:39.984440 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:43:39.984446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:43:39.984728 (+ 282us) client_negotiation.cc:769] Sending connection context
0619 14:43:39.984886 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 14:43:39.985143 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":110,"threads_started":1}
I20260619 14:43:39.987381 11128 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39685
I20260619 14:43:39.987697 11128 heartbeater.cc:464] Registering TS with master...
I20260619 14:43:39.988279 11128 heartbeater.cc:511] Master 127.0.98.62:39685 requested a full tablet report, sending...
I20260619 14:43:39.989415 10669 ts_manager.cc:194] Registered new tserver with Master: b43cb7c8e49049849e6066333ff5f011 (127.0.98.3:36325)
I20260619 14:43:39.989956 10669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52013
I20260619 14:43:40.001057 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:43:40.011891 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:40.003357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.0.98.62:39685)
0619 14:43:40.003588 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:40.003593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:40.003750 (+ 157us) server_negotiation.cc:407] Connection header received
0619 14:43:40.003914 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:40.003918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:40.003969 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:40.004058 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:43:40.004786 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.005354 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.006228 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.006451 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.006563 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:40.008183 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:43:40.008186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:43:40.008188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:43:40.008212 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:43:40.009925 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:40.010526 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:40.010530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:40.010532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:40.010619 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:43:40.010912 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:43:40.010917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:43:40.010919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:43:40.011179 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:43:40.011324 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:40.011587 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 14:43:40.011720 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:43:40.015046 10669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:43:40.018170 10669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:43:40.034116 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:40.028566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.0.98.2:45019)
0619 14:43:40.028915 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:40.028927 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:40.029417 (+ 490us) server_negotiation.cc:407] Connection header received
0619 14:43:40.029487 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:40.029492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:40.029648 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:40.029788 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:40.030225 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.031263 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.032789 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.033420 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.033524 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:40.033663 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:40.033859 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 14:43:40.033965 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":138,"threads_started":1}
I20260619 14:43:40.034091 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:40.027459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34311 (local address 127.0.0.1:49712)
0619 14:43:40.028270 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:40.028334 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:40.028480 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:40.029115 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:40.029121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:40.029143 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:40.029371 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:40.029379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.030685 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.030689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:40.032271 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:40.032280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.033573 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.033577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:40.033800 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:40.033827 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:43:40.033886 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 14:43:40.033942 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":152,"spinlock_wait_cycles":896,"thread_start_us":96,"threads_started":1}
I20260619 14:43:40.034235 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:40.028604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38674 (local address 127.0.98.3:36325)
0619 14:43:40.029339 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:40.029354 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:40.029378 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:43:40.029455 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:40.029460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:40.029638 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:40.029788 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:40.030020 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.030921 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.032443 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.033289 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.033383 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:40.033507 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:40.033975 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 14:43:40.034076 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":127,"threads_started":1}
I20260619 14:43:40.034437 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:40.027585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.0.98.1:34311)
0619 14:43:40.027961 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:40.027967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:40.028628 (+ 661us) server_negotiation.cc:407] Connection header received
0619 14:43:40.028719 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:40.028726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:40.028944 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:40.029127 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:40.029491 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.030568 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.032392 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.033223 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.033349 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:40.033507 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:40.034181 (+ 674us) server_negotiation.cc:299] Negotiation successful
0619 14:43:40.034297 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":160,"threads_started":1}
I20260619 14:43:40.034091 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:40.028320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45019 (local address 127.0.0.1:36346)
0619 14:43:40.029217 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:40.029231 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:40.029323 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:40.029991 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:40.029994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:40.030014 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:40.030102 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:40.030108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.031601 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.031605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:40.032677 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:40.032684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.033573 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.033577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:40.033713 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:40.033736 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:43:40.033815 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 14:43:40.033901 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":37,"thread_start_us":100,"threads_started":1}
I20260619 14:43:40.034722 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:40.027515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36325 (local address 127.0.0.1:38674)
0619 14:43:40.028270 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:40.028334 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:40.028883 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:40.029788 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:40.029791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:40.029809 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:40.029906 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:40.029912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.031061 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.031064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:40.032316 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:40.032326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.033407 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.033411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:40.033534 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:40.033572 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 14:43:40.033695 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 14:43:40.033835 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":26,"spinlock_wait_cycles":9088,"thread_start_us":104,"threads_started":1}
I20260619 14:43:40.037559 10928 tablet_service.cc:1467] Processing CreateTablet for tablet 9af69001d80f4b72a7cb6d88a93bc1ae (DEFAULT_TABLE table=test-table [id=d3c09e78b20943f1800532b9d7625893]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:40.037607 11062 tablet_service.cc:1467] Processing CreateTablet for tablet 9af69001d80f4b72a7cb6d88a93bc1ae (DEFAULT_TABLE table=test-table [id=d3c09e78b20943f1800532b9d7625893]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:40.037607 10793 tablet_service.cc:1467] Processing CreateTablet for tablet 9af69001d80f4b72a7cb6d88a93bc1ae (DEFAULT_TABLE table=test-table [id=d3c09e78b20943f1800532b9d7625893]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:43:40.038549 10928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af69001d80f4b72a7cb6d88a93bc1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:40.038600 11062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af69001d80f4b72a7cb6d88a93bc1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:40.038594 10793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af69001d80f4b72a7cb6d88a93bc1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:43:40.045446 11145 tablet_bootstrap.cc:492] T 9af69001d80f4b72a7cb6d88a93bc1ae P 84a710f2d902404791f2a0fe5aa6b0a6: Bootstrap starting.
I20260619 14:43:40.046170 11146 tablet_bootstrap.cc:492] T 9af69001d80f4b72a7cb6d88a93bc1ae P ca198a1372434e2b82f2b384f9e916df: Bootstrap starting.
I20260619 14:43:40.046573 11147 tablet_bootstrap.cc:492] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011: Bootstrap starting.
I20260619 14:43:40.049163 11147 tablet_bootstrap.cc:654] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:40.049338 11145 tablet_bootstrap.cc:654] T 9af69001d80f4b72a7cb6d88a93bc1ae P 84a710f2d902404791f2a0fe5aa6b0a6: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:40.049563 11146 tablet_bootstrap.cc:654] T 9af69001d80f4b72a7cb6d88a93bc1ae P ca198a1372434e2b82f2b384f9e916df: Neither blocks nor log segments found. Creating new log.
I20260619 14:43:40.050292 11146 log.cc:824] T 9af69001d80f4b72a7cb6d88a93bc1ae P ca198a1372434e2b82f2b384f9e916df: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:40.050359 11145 log.cc:824] T 9af69001d80f4b72a7cb6d88a93bc1ae P 84a710f2d902404791f2a0fe5aa6b0a6: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:40.050431 11147 log.cc:824] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011: Log is configured to *not* fsync() on all Append() calls
I20260619 14:43:40.052305 11147 tablet_bootstrap.cc:492] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011: No bootstrap required, opened a new log
I20260619 14:43:40.052373 11145 tablet_bootstrap.cc:492] T 9af69001d80f4b72a7cb6d88a93bc1ae P 84a710f2d902404791f2a0fe5aa6b0a6: No bootstrap required, opened a new log
I20260619 14:43:40.052373 11146 tablet_bootstrap.cc:492] T 9af69001d80f4b72a7cb6d88a93bc1ae P ca198a1372434e2b82f2b384f9e916df: No bootstrap required, opened a new log
I20260619 14:43:40.052539 11147 ts_tablet_manager.cc:1397] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:43:40.052544 11145 ts_tablet_manager.cc:1397] T 9af69001d80f4b72a7cb6d88a93bc1ae P 84a710f2d902404791f2a0fe5aa6b0a6: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260619 14:43:40.052544 11146 ts_tablet_manager.cc:1397] T 9af69001d80f4b72a7cb6d88a93bc1ae P ca198a1372434e2b82f2b384f9e916df: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 14:43:40.055568 11147 raft_consensus.cc:348] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca198a1372434e2b82f2b384f9e916df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } } peers { permanent_uuid: "b43cb7c8e49049849e6066333ff5f011" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36325 } } peers { permanent_uuid: "84a710f2d902404791f2a0fe5aa6b0a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45019 } }
I20260619 14:43:40.056015 11147 raft_consensus.cc:374] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:40.056432 11147 raft_consensus.cc:729] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b43cb7c8e49049849e6066333ff5f011, State: Initialized, Role: FOLLOWER
I20260619 14:43:40.056612 11145 raft_consensus.cc:348] T 9af69001d80f4b72a7cb6d88a93bc1ae P 84a710f2d902404791f2a0fe5aa6b0a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca198a1372434e2b82f2b384f9e916df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } } peers { permanent_uuid: "b43cb7c8e49049849e6066333ff5f011" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36325 } } peers { permanent_uuid: "84a710f2d902404791f2a0fe5aa6b0a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45019 } }
I20260619 14:43:40.056936 11145 raft_consensus.cc:374] T 9af69001d80f4b72a7cb6d88a93bc1ae P 84a710f2d902404791f2a0fe5aa6b0a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:40.057025 11145 raft_consensus.cc:729] T 9af69001d80f4b72a7cb6d88a93bc1ae P 84a710f2d902404791f2a0fe5aa6b0a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84a710f2d902404791f2a0fe5aa6b0a6, State: Initialized, Role: FOLLOWER
I20260619 14:43:40.057039 11147 consensus_queue.cc:260] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca198a1372434e2b82f2b384f9e916df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } } peers { permanent_uuid: "b43cb7c8e49049849e6066333ff5f011" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36325 } } peers { permanent_uuid: "84a710f2d902404791f2a0fe5aa6b0a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45019 } }
I20260619 14:43:40.057121 11146 raft_consensus.cc:348] T 9af69001d80f4b72a7cb6d88a93bc1ae P ca198a1372434e2b82f2b384f9e916df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca198a1372434e2b82f2b384f9e916df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } } peers { permanent_uuid: "b43cb7c8e49049849e6066333ff5f011" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36325 } } peers { permanent_uuid: "84a710f2d902404791f2a0fe5aa6b0a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45019 } }
I20260619 14:43:40.057458 11146 raft_consensus.cc:374] T 9af69001d80f4b72a7cb6d88a93bc1ae P ca198a1372434e2b82f2b384f9e916df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:43:40.057528 11146 raft_consensus.cc:729] T 9af69001d80f4b72a7cb6d88a93bc1ae P ca198a1372434e2b82f2b384f9e916df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca198a1372434e2b82f2b384f9e916df, State: Initialized, Role: FOLLOWER
I20260619 14:43:40.057495 11145 consensus_queue.cc:260] T 9af69001d80f4b72a7cb6d88a93bc1ae P 84a710f2d902404791f2a0fe5aa6b0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca198a1372434e2b82f2b384f9e916df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } } peers { permanent_uuid: "b43cb7c8e49049849e6066333ff5f011" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36325 } } peers { permanent_uuid: "84a710f2d902404791f2a0fe5aa6b0a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45019 } }
I20260619 14:43:40.058065 11146 consensus_queue.cc:260] T 9af69001d80f4b72a7cb6d88a93bc1ae P ca198a1372434e2b82f2b384f9e916df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca198a1372434e2b82f2b384f9e916df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } } peers { permanent_uuid: "b43cb7c8e49049849e6066333ff5f011" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36325 } } peers { permanent_uuid: "84a710f2d902404791f2a0fe5aa6b0a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45019 } }
I20260619 14:43:40.058228 11147 ts_tablet_manager.cc:1428] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:43:40.058400 10994 heartbeater.cc:503] Master 127.0.98.62:39685 was elected leader, sending a full tablet report...
I20260619 14:43:40.058529 11128 heartbeater.cc:503] Master 127.0.98.62:39685 was elected leader, sending a full tablet report...
I20260619 14:43:40.058684 11145 ts_tablet_manager.cc:1428] T 9af69001d80f4b72a7cb6d88a93bc1ae P 84a710f2d902404791f2a0fe5aa6b0a6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:43:40.059043 11146 ts_tablet_manager.cc:1428] T 9af69001d80f4b72a7cb6d88a93bc1ae P ca198a1372434e2b82f2b384f9e916df: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:43:40.059362 10860 heartbeater.cc:503] Master 127.0.98.62:39685 was elected leader, sending a full tablet report...
I20260619 14:43:40.120380 11151 raft_consensus.cc:482] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:43:40.120594 11151 raft_consensus.cc:504] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca198a1372434e2b82f2b384f9e916df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } } peers { permanent_uuid: "b43cb7c8e49049849e6066333ff5f011" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36325 } } peers { permanent_uuid: "84a710f2d902404791f2a0fe5aa6b0a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45019 } }
I20260619 14:43:40.121793 11151 leader_election.cc:288] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca198a1372434e2b82f2b384f9e916df (127.0.98.1:34311), 84a710f2d902404791f2a0fe5aa6b0a6 (127.0.98.2:45019)
I20260619 14:43:40.125922 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:40.122004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34311 (local address 127.0.98.3:52497)
0619 14:43:40.122153 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:40.122170 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:40.122277 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:40.122609 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:40.122612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:40.122631 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:40.122837 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:40.122848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.123859 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.123865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:40.124692 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:40.124700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.125479 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.125483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:40.125592 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:40.125610 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:43:40.125699 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 14:43:40.125775 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":22}
I20260619 14:43:40.125922 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:40.122221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52497 (local address 127.0.98.1:34311)
0619 14:43:40.122346 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:40.122351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:40.122368 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:43:40.122435 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:40.122437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:40.122490 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:40.122594 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:40.123181 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.123731 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.124830 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.125345 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.125464 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:40.125532 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:40.125716 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 14:43:40.125800 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:43:40.126395 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:40.122249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38753 (local address 127.0.98.2:45019)
0619 14:43:40.122362 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:40.122367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:40.122610 (+ 243us) server_negotiation.cc:407] Connection header received
0619 14:43:40.122926 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:40.122929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:40.123043 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:40.123161 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:43:40.123744 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.124236 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.125315 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.125827 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.125950 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:40.126008 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:40.126238 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 14:43:40.126308 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:43:40.126791 11154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:40.122141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45019 (local address 127.0.98.3:38753)
0619 14:43:40.122497 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:40.122507 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:43:40.122619 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:43:40.123171 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:43:40.123173 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:43:40.123305 (+ 132us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:43:40.123482 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:40.123491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.124354 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.124358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:40.125190 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:43:40.125198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.125962 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.125965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:43:40.126077 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:40.126109 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 14:43:40.126560 (+ 451us) client_negotiation.cc:240] Negotiation successful
0619 14:43:40.126666 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":21,"thread_start_us":138,"threads_started":1}
I20260619 14:43:40.126771 10814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af69001d80f4b72a7cb6d88a93bc1ae" candidate_uuid: "b43cb7c8e49049849e6066333ff5f011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca198a1372434e2b82f2b384f9e916df" is_pre_election: true
I20260619 14:43:40.127131 10814 raft_consensus.cc:2454] T 9af69001d80f4b72a7cb6d88a93bc1ae P ca198a1372434e2b82f2b384f9e916df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b43cb7c8e49049849e6066333ff5f011 in term 0.
I20260619 14:43:40.127563 10948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af69001d80f4b72a7cb6d88a93bc1ae" candidate_uuid: "b43cb7c8e49049849e6066333ff5f011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84a710f2d902404791f2a0fe5aa6b0a6" is_pre_election: true
I20260619 14:43:40.127704 11015 leader_election.cc:302] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b43cb7c8e49049849e6066333ff5f011, ca198a1372434e2b82f2b384f9e916df; no voters:
I20260619 14:43:40.127909 10948 raft_consensus.cc:2454] T 9af69001d80f4b72a7cb6d88a93bc1ae P 84a710f2d902404791f2a0fe5aa6b0a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b43cb7c8e49049849e6066333ff5f011 in term 0.
I20260619 14:43:40.128010 11151 raft_consensus.cc:2781] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:43:40.128096 11151 raft_consensus.cc:482] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:43:40.128157 11151 raft_consensus.cc:3037] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:40.129200 11151 raft_consensus.cc:504] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca198a1372434e2b82f2b384f9e916df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } } peers { permanent_uuid: "b43cb7c8e49049849e6066333ff5f011" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36325 } } peers { permanent_uuid: "84a710f2d902404791f2a0fe5aa6b0a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45019 } }
I20260619 14:43:40.129613 11151 leader_election.cc:288] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [CANDIDATE]: Term 1 election: Requested vote from peers ca198a1372434e2b82f2b384f9e916df (127.0.98.1:34311), 84a710f2d902404791f2a0fe5aa6b0a6 (127.0.98.2:45019)
I20260619 14:43:40.130038 10814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af69001d80f4b72a7cb6d88a93bc1ae" candidate_uuid: "b43cb7c8e49049849e6066333ff5f011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca198a1372434e2b82f2b384f9e916df"
I20260619 14:43:40.130061 10948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af69001d80f4b72a7cb6d88a93bc1ae" candidate_uuid: "b43cb7c8e49049849e6066333ff5f011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84a710f2d902404791f2a0fe5aa6b0a6"
I20260619 14:43:40.130182 10948 raft_consensus.cc:3037] T 9af69001d80f4b72a7cb6d88a93bc1ae P 84a710f2d902404791f2a0fe5aa6b0a6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:40.130199 10814 raft_consensus.cc:3037] T 9af69001d80f4b72a7cb6d88a93bc1ae P ca198a1372434e2b82f2b384f9e916df [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:43:40.131522 10948 raft_consensus.cc:2454] T 9af69001d80f4b72a7cb6d88a93bc1ae P 84a710f2d902404791f2a0fe5aa6b0a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b43cb7c8e49049849e6066333ff5f011 in term 1.
I20260619 14:43:40.131641 10814 raft_consensus.cc:2454] T 9af69001d80f4b72a7cb6d88a93bc1ae P ca198a1372434e2b82f2b384f9e916df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b43cb7c8e49049849e6066333ff5f011 in term 1.
I20260619 14:43:40.131837 11018 leader_election.cc:302] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84a710f2d902404791f2a0fe5aa6b0a6, b43cb7c8e49049849e6066333ff5f011; no voters:
I20260619 14:43:40.132009 11151 raft_consensus.cc:2781] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:43:40.132267 11151 raft_consensus.cc:686] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [term 1 LEADER]: Becoming Leader. State: Replica: b43cb7c8e49049849e6066333ff5f011, State: Running, Role: LEADER
I20260619 14:43:40.132602 11151 consensus_queue.cc:237] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca198a1372434e2b82f2b384f9e916df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } } peers { permanent_uuid: "b43cb7c8e49049849e6066333ff5f011" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36325 } } peers { permanent_uuid: "84a710f2d902404791f2a0fe5aa6b0a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45019 } }
I20260619 14:43:40.135879 10668 catalog_manager.cc:5495] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 reported cstate change: term changed from 0 to 1, leader changed from <none> to b43cb7c8e49049849e6066333ff5f011 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b43cb7c8e49049849e6066333ff5f011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca198a1372434e2b82f2b384f9e916df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b43cb7c8e49049849e6066333ff5f011" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84a710f2d902404791f2a0fe5aa6b0a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45019 } health_report { overall_health: UNKNOWN } } }
I20260619 14:43:40.163676 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:40.159882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38684 (local address 127.0.98.3:36325)
0619 14:43:40.160018 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:43:40.160023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:43:40.160038 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:43:40.160175 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:43:40.160178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:43:40.160243 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:43:40.160362 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:43:40.160623 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.161126 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.162083 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:43:40.162344 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:43:40.162453 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:43:40.162778 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:43:40.163191 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:43:40.163335 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 14:43:40.163451 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 14:43:40.163528 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:43:40.169998 10814 raft_consensus.cc:1261] T 9af69001d80f4b72a7cb6d88a93bc1ae P ca198a1372434e2b82f2b384f9e916df [term 1 FOLLOWER]: Refusing update from remote peer b43cb7c8e49049849e6066333ff5f011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:40.169998 10948 raft_consensus.cc:1261] T 9af69001d80f4b72a7cb6d88a93bc1ae P 84a710f2d902404791f2a0fe5aa6b0a6 [term 1 FOLLOWER]: Refusing update from remote peer b43cb7c8e49049849e6066333ff5f011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:43:40.170787 11151 consensus_queue.cc:1035] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84a710f2d902404791f2a0fe5aa6b0a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:40.170990 11155 consensus_queue.cc:1035] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca198a1372434e2b82f2b384f9e916df" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:43:40.181651 11160 mvcc.cc:205] Tried to move back new op lower bound from 7298581381808533504 to 7298581381666099200. Current Snapshot: MvccSnapshot[applied={T|T < 7298581381808533504 or (T in {7298581381808533504})}]
I20260619 14:43:40.186908 10667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:43:40.187088 10667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:43:40.189539 10667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46be1f8b797e43b7a4988b3f3fbe99e0: Sending DeleteTablet for 3 replicas of tablet 9af69001d80f4b72a7cb6d88a93bc1ae
I20260619 14:43:40.190341 10793 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af69001d80f4b72a7cb6d88a93bc1ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49712
I20260619 14:43:40.190367 11062 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af69001d80f4b72a7cb6d88a93bc1ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38674
I20260619 14:43:40.190977 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 10731
I20260619 14:43:40.191006 11167 tablet_replica.cc:330] T 9af69001d80f4b72a7cb6d88a93bc1ae P ca198a1372434e2b82f2b384f9e916df: stopping tablet replica
I20260619 14:43:40.191164 10928 tablet_service.cc:1514] Processing DeleteTablet for tablet 9af69001d80f4b72a7cb6d88a93bc1ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:43:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36346
I20260619 14:43:40.199693 11169 tablet_replica.cc:330] T 9af69001d80f4b72a7cb6d88a93bc1ae P 84a710f2d902404791f2a0fe5aa6b0a6: stopping tablet replica
I20260619 14:43:40.200066 11168 tablet_replica.cc:330] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011: stopping tablet replica
I20260619 14:43:40.200110 11169 raft_consensus.cc:2229] T 9af69001d80f4b72a7cb6d88a93bc1ae P 84a710f2d902404791f2a0fe5aa6b0a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:43:40.200420 11168 raft_consensus.cc:2229] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:43:40.200459 11169 raft_consensus.cc:2258] T 9af69001d80f4b72a7cb6d88a93bc1ae P 84a710f2d902404791f2a0fe5aa6b0a6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 14:43:40.200843 10653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34311 (error 108)
I20260619 14:43:40.200917 11168 raft_consensus.cc:2258] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:43:40.201720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 10865
I20260619 14:43:40.204442 11168 ts_tablet_manager.cc:1905] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:43:40.205932 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:40.205624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34311 (local address 127.0.0.1:49726)
0619 14:43:40.205778 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:40.205849 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 14:43:40.206085 10653 catalog_manager.cc:4584] TS ca198a1372434e2b82f2b384f9e916df (127.0.98.1:34311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9af69001d80f4b72a7cb6d88a93bc1ae: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34311: connect: Connection refused (error 111)
I20260619 14:43:40.207898 11168 ts_tablet_manager.cc:1918] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:43:40.208003 11168 log.cc:1196] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/9af69001d80f4b72a7cb6d88a93bc1ae
I20260619 14:43:40.208351 11168 ts_tablet_manager.cc:1939] T 9af69001d80f4b72a7cb6d88a93bc1ae P b43cb7c8e49049849e6066333ff5f011: Deleting consensus metadata
I20260619 14:43:40.209652 10655 catalog_manager.cc:4841] TS b43cb7c8e49049849e6066333ff5f011 (127.0.98.3:36325): tablet 9af69001d80f4b72a7cb6d88a93bc1ae (table test-table [id=d3c09e78b20943f1800532b9d7625893]) successfully deleted
I20260619 14:43:40.209789 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:43:40.209512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45019 (local address 127.0.0.1:36358)
0619 14:43:40.209637 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 14:43:40.209698 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 14:43:40.209954 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 10999
I20260619 14:43:40.215633 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 10637
2026-06-19T14:43:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2405 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:43:40 dist-test-slave-cr22 krb5kdc[11173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:43:40 dist-test-slave-cr22 krb5kdc[11173](info): set up 2 sockets
Jun 19 14:43:40 dist-test-slave-cr22 krb5kdc[11173](info): commencing operation
krb5kdc: starting...
W20260619 14:43:41.308588 392 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:43:41 dist-test-slave-cr22 krb5kdc[11173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:43:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:43:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 14:43:41.496989 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZ9bYsM/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/postgres -l logfile start
2026-06-19 14:43:44.263 UTC [11203] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 14:43:44.263 UTC [11203] LOG: listening on IPv4 address "127.0.98.20", port 52673
2026-06-19 14:43:44.267 UTC [11203] LOG: listening on Unix socket "/tmp/.s.PGSQL.52673"
2026-06-19 14:43:44.304 UTC [11206] LOG: database system was shut down at 2026-06-19 14:43:42 UTC
2026-06-19 14:43:44.308 UTC [11203] LOG: database system is ready to accept connections
I20260619 14:43:45.280428 392 mini_postgres.cc:96] Postgres bound to 52673
2026-06-19 14:43:45.287 UTC [11214] FATAL: database "slave" does not exist
127.0.98.20:52673 - accepting connections
I20260619 14:43:45.288038 392 mini_ranger.cc:161] Starting Ranger...
I20260619 14:43:45.303975 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 14:43:45.852588 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 14:43:45.852680 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ranger-admin
2026-06-19 14:43:46,142 [I] DB FLAVOR :POSTGRES
2026-06-19 14:43:46,142 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:43:46,142 [I] Checking connection..
2026-06-19 14:43:46,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 14:43:46,422 [I] Checking connection passed.
2026-06-19 14:43:46,422 [I] --------- Verifying version history table ---------
2026-06-19 14:43:46,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:43:46,670 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:43:46,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:43:46,925 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:43:46,925 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 14:43:46,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 14:43:47.178 UTC [11278] WARNING: there is no transaction in progress
2026-06-19 14:43:47,185 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 14:43:47,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:43:47,442 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 14:43:47,442 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:43:47,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 14:43:47,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 14:43:47,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cr22.c.gcp-upstream.internal','N') ;"
2026-06-19 14:43:48,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:43:48,440 [I] Table x_portal_user does not exist in database ranger
2026-06-19 14:43:48,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:43:48,694 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 14:43:48,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:43:48,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:43:49,225 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 14:43:49,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 14:43:50.297 UTC [11386] WARNING: there is no transaction in progress
2026-06-19 14:43:50.317 UTC [11386] WARNING: there is no transaction in progress
2026-06-19 14:43:50.336 UTC [11386] WARNING: there is no transaction in progress
2026-06-19 14:43:50.355 UTC [11386] WARNING: there is no transaction in progress
2026-06-19 14:43:50.374 UTC [11386] WARNING: there is no transaction in progress
2026-06-19 14:43:50.392 UTC [11386] WARNING: there is no transaction in progress
2026-06-19 14:43:50.450 UTC [11386] WARNING: there is no transaction in progress
2026-06-19 14:43:50.458 UTC [11386] WARNING: there is no transaction in progress
2026-06-19 14:43:50.475 UTC [11386] WARNING: there is no transaction in progress
2026-06-19 14:43:50.487 UTC [11386] WARNING: there is no transaction in progress
2026-06-19 14:43:50.499 UTC [11386] WARNING: there is no transaction in progress
2026-06-19 14:43:50.517 UTC [11386] WARNING: there is no transaction in progress
2026-06-19 14:43:50.528 UTC [11386] WARNING: there is no transaction in progress
2026-06-19 14:43:50.535 UTC [11386] WARNING: there is no transaction in progress
2026-06-19 14:43:50.544 UTC [11386] WARNING: there is no transaction in progress
2026-06-19 14:43:50.552 UTC [11386] WARNING: there is no transaction in progress
2026-06-19 14:43:51,343 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 14:43:51,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 14:43:51,603 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 14:43:51,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:43:51,862 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 14:43:51,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:43:52,118 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 14:43:52,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:43:52,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:43:52,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cr22.c.gcp-upstream.internal';"
2026-06-19 14:43:52,921 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 14:43:52,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 14:43:53,170 [I] DB_PATCHES have already been applied
I20260619 14:43:53.175765 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZ9bYsM/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 14:43:53.175833 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 14:43:53.179256 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:36693
Jun 19, 2026 2:43:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:43:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:43:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:43:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:43:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:43:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:43:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 14:43:54.320861 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.033s user 0.005s sys 0.005s
Jun 19, 2026 2:44:15 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:44:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Jun 19, 2026 2:44:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:44:16 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:44:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:44:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:44:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:44:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:44:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:44:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:44:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:44:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:44:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:44:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:44:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:44:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:44:16 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:44:16 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 14:44:17.574549 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 14:44:17 dist-test-slave-cr22 krb5kdc[11173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880257, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 14:44:17.668583 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 14:44:17.679032 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 14:44:18.691203 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 14:44:18.691298 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ranger-kms
I20260619 14:44:18.691398 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:52673
2026-06-19 14:44:18,887 [I] DB FLAVOR :POSTGRES
2026-06-19 14:44:18,887 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:44:18,887 [I] Checking connection
2026-06-19 14:44:18,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 14:44:19,155 [I] connection success
2026-06-19 14:44:19,156 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 14:44:19,156 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 14:44:19,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 14:44:19,406 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 14:44:19,406 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:44:19,406 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 14:44:19,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52673/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 14:44:19,680 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 14:44:20.210767 392 mini_ranger_kms.cc:310] Created kms service
I20260619 14:44:20.401921 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 14:44:20.450423 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 14:44:20 dist-test-slave-cr22 krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880257, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:44:20.868462 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 14:44:20.868623 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 14:44:20.868645 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 14:44:20.872876 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11567
I20260619 14:44:20.873013 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:34911
Jun 19, 2026 2:44:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:44:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:44:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:44:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:44:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:44:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 14:44:23 dist-test-slave-cr22 krb5kdc[11173](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880263, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:44:23.171969 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.503s user 0.005s sys 0.174s
I20260619 14:44:23.172148 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 14:44:23.172334 392 mini_ranger_kms.cc:401] 127.0.98.20:34911/kms/v1/keys
Jun 19 14:44:23 dist-test-slave-cr22 krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880263, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 14:44:23 dist-test-slave-cr22 krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880263, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 14:44:26 dist-test-slave-cr22 krb5kdc[11173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:44:26.397034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45229
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45229
--ranger_config_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34911/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:44:26.535843 11617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:44:26.536129 11617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:44:26.536201 11617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:44:26.540912 11617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:44:26.540984 11617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:44:26.541004 11617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:44:26.541021 11617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:44:26.541038 11617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:44:26.546792 11617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:34911/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45229
--ranger_config_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:44:26.547979 11617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:44:26.549011 11617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:44:26.558634 11622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:44:26.558833 11623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:44:26.558717 11625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:44:26.559010 11617 server_base.cc:1034] running on GCE node
I20260619 14:44:26.559617 11617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:44:26.560770 11617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:44:26.561956 11617 hybrid_clock.cc:648] HybridClock initialized: now 1781880266561931 us; error 45 us; skew 500 ppm
Jun 19 14:44:26 dist-test-slave-cr22 krb5kdc[11173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:44:26.571694 11617 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:44:26.572820 11617 webserver.cc:460] Webserver started at http://127.0.98.62:35033/ using document root <none> and password file <none>
I20260619 14:44:26.573371 11617 fs_manager.cc:359] Metadata directory not provided
I20260619 14:44:26.573427 11617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:44:26.573655 11617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:44:26 dist-test-slave-cr22 krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:44:26.636897 11617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "ed388cdcc60f492bab90574ec9e2ec47"
format_stamp: "Formatted at 2026-06-19 14:44:26 on dist-test-slave-cr22"
server_key: "7531b6ff6e43a2d64e5d3a1628f7f3f4"
server_key_iv: "c84a5efe822403eef3374fa1bc6d6dd0"
server_key_version: "kuduclusterkey@0"
I20260619 14:44:26.637549 11617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "ed388cdcc60f492bab90574ec9e2ec47"
format_stamp: "Formatted at 2026-06-19 14:44:26 on dist-test-slave-cr22"
server_key: "7531b6ff6e43a2d64e5d3a1628f7f3f4"
server_key_iv: "c84a5efe822403eef3374fa1bc6d6dd0"
server_key_version: "kuduclusterkey@0"
I20260619 14:44:26.641798 11617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:44:26.659955 11632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:44:26.661970 11617 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 14:44:26.662175 11617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "ed388cdcc60f492bab90574ec9e2ec47"
format_stamp: "Formatted at 2026-06-19 14:44:26 on dist-test-slave-cr22"
server_key: "7531b6ff6e43a2d64e5d3a1628f7f3f4"
server_key_iv: "c84a5efe822403eef3374fa1bc6d6dd0"
server_key_version: "kuduclusterkey@0"
I20260619 14:44:26.662328 11617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:44:26.682725 11617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:44:26.686322 11617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:44:26.686533 11617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:44:26.695736 11617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45229
I20260619 14:44:26.695770 11684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45229 every 8 connection(s)
I20260619 14:44:26.696971 11617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:44:26.700031 11685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:44:26.705904 11685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47: Bootstrap starting.
I20260619 14:44:26.706532 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 11617
I20260619 14:44:26.706663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:44:26.706948 392 external_mini_cluster.cc:1428] Setting key 5f1b9cd5446988fc6477103c02ddd9de
I20260619 14:44:26.708948 11685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47: Neither blocks nor log segments found. Creating new log.
I20260619 14:44:26.709709 11685 log.cc:824] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47: Log is configured to *not* fsync() on all Append() calls
I20260619 14:44:26.712069 11685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47: No bootstrap required, opened a new log
I20260619 14:44:26.716573 11685 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed388cdcc60f492bab90574ec9e2ec47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45229 } }
I20260619 14:44:26.716912 11685 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:44:26.717010 11685 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed388cdcc60f492bab90574ec9e2ec47, State: Initialized, Role: FOLLOWER
Jun 19 14:44:26 dist-test-slave-cr22 krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:44:26.717587 11685 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed388cdcc60f492bab90574ec9e2ec47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45229 } }
I20260619 14:44:26.717824 11685 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:44:26.717912 11685 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:44:26.718019 11685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:44:26.719164 11685 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed388cdcc60f492bab90574ec9e2ec47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45229 } }
I20260619 14:44:26.719575 11685 leader_election.cc:302] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed388cdcc60f492bab90574ec9e2ec47; no voters:
I20260619 14:44:26.719981 11685 leader_election.cc:288] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:44:26.720078 11690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:44:26.720322 11690 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47 [term 1 LEADER]: Becoming Leader. State: Replica: ed388cdcc60f492bab90574ec9e2ec47, State: Running, Role: LEADER
I20260619 14:44:26.720670 11690 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed388cdcc60f492bab90574ec9e2ec47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45229 } }
I20260619 14:44:26.721457 11685 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:44:26.721909 11690 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed388cdcc60f492bab90574ec9e2ec47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed388cdcc60f492bab90574ec9e2ec47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45229 } } }
I20260619 14:44:26.722095 11690 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47 [sys.catalog]: This master's current role is: LEADER
I20260619 14:44:26.722362 11690 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed388cdcc60f492bab90574ec9e2ec47. Latest consensus state: current_term: 1 leader_uuid: "ed388cdcc60f492bab90574ec9e2ec47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed388cdcc60f492bab90574ec9e2ec47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45229 } } }
I20260619 14:44:26.722520 11690 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47 [sys.catalog]: This master's current role is: LEADER
I20260619 14:44:26.722846 11695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:44:26.724330 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:26.708394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49556 (local address 127.0.98.62:45229)
0619 14:44:26.708862 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0619 14:44:26.708881 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:44:26.708924 (+ 43us) server_negotiation.cc:407] Connection header received
0619 14:44:26.709785 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:44:26.709811 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:44:26.710253 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:44:26.710692 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:44:26.711566 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:26.713056 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:26.714353 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:26.715042 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:26.715209 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:44:26.718278 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:44:26.718284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:44:26.718303 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:44:26.718333 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:44:26.720355 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:44:26.720875 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:44:26.720882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:44:26.720888 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:44:26.721010 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:44:26.721322 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:44:26.721324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:44:26.721326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:44:26.721819 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:44:26.722693 (+ 874us) server_negotiation.cc:1035] Waiting for connection context
0619 14:44:26.722851 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 14:44:26.723363 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":124,"threads_started":1}
I20260619 14:44:26.726539 11695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:44:26.727100 11685 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:44:26.739872 11695 catalog_manager.cc:1347] Generated new cluster ID: 109f845f01614a8dbe3f20ecbb6f954b
I20260619 14:44:26.739957 11695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:44:26.751493 11695 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:44:26.752966 11695 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:44:26.764576 11695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47: Generated new TSK 0
I20260619 14:44:26.765952 11695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:44:27.547583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34911/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45229
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:44:27.690922 11730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:44:27.691226 11730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:44:27.691298 11730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:44:27.697356 11730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:44:27.697491 11730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:44:27.697610 11730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:44:27.704500 11730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:34911/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:44:27.705633 11730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:44:27.706710 11730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:44:27.718312 11737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:44:27.718468 11738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:44:27.718331 11740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:44:27.719457 11730 server_base.cc:1034] running on GCE node
I20260619 14:44:27.719951 11730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:44:27.720748 11730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:44:27.721961 11730 hybrid_clock.cc:648] HybridClock initialized: now 1781880267721935 us; error 38 us; skew 500 ppm
Jun 19 14:44:27 dist-test-slave-cr22 krb5kdc[11173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:44:27.725728 11730 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:44:27.726922 11730 webserver.cc:460] Webserver started at http://127.0.98.1:39909/ using document root <none> and password file <none>
I20260619 14:44:27.727710 11730 fs_manager.cc:359] Metadata directory not provided
I20260619 14:44:27.727854 11730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:44:27.728144 11730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:44:27 dist-test-slave-cr22 krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:44:27.745724 11730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "7cc439f6a90c4660aa524ec20cbe37a2"
format_stamp: "Formatted at 2026-06-19 14:44:27 on dist-test-slave-cr22"
server_key: "8f8bf2cfd8fc1689e3dae33ea5910758"
server_key_iv: "2bf692715742fcb5d70f2ef38d4fc08b"
server_key_version: "kuduclusterkey@0"
I20260619 14:44:27.746644 11730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7cc439f6a90c4660aa524ec20cbe37a2"
format_stamp: "Formatted at 2026-06-19 14:44:27 on dist-test-slave-cr22"
server_key: "8f8bf2cfd8fc1689e3dae33ea5910758"
server_key_iv: "2bf692715742fcb5d70f2ef38d4fc08b"
server_key_version: "kuduclusterkey@0"
I20260619 14:44:27.752214 11730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:44:27.769153 11747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:44:27.770936 11730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:44:27.771066 11730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "7cc439f6a90c4660aa524ec20cbe37a2"
format_stamp: "Formatted at 2026-06-19 14:44:27 on dist-test-slave-cr22"
server_key: "8f8bf2cfd8fc1689e3dae33ea5910758"
server_key_iv: "2bf692715742fcb5d70f2ef38d4fc08b"
server_key_version: "kuduclusterkey@0"
I20260619 14:44:27.771211 11730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:44:27.803975 11730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:44:27.808239 11730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:44:27.808524 11730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:44:27.809252 11730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:44:27.810335 11730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:44:27.810436 11730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:44:27.810559 11730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:44:27.810636 11730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:44:27.821065 11730 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40649
I20260619 14:44:27.822413 11730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:44:27.825263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 11730
I20260619 14:44:27.825384 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:44:27.825687 392 external_mini_cluster.cc:1428] Setting key a5a1d8e5f2d63ca3c9f0c9148fbb2d72
I20260619 14:44:27.832832 11862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40649 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Jun 19 14:44:27 dist-test-slave-cr22 krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:44:27.853116 11867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:27.835345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45229 (local address 127.0.98.1:48651)
0619 14:44:27.836922 (+ 1577us) negotiation.cc:107] Waiting for socket to connect
0619 14:44:27.836975 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:44:27.838030 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:44:27.839037 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:44:27.839046 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:44:27.839397 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:44:27.840059 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:44:27.840076 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:44:27.841229 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:44:27.841233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:44:27.842623 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:44:27.842634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:44:27.843379 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:44:27.843382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:44:27.843591 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:44:27.844275 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:44:27.844294 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:44:27.846011 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:44:27.849880 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:44:27.849886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:44:27.849903 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:44:27.850271 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:44:27.850569 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:44:27.850572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:44:27.850574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:44:27.850751 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:44:27.851219 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:44:27.851225 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:44:27.851570 (+ 345us) client_negotiation.cc:769] Sending connection context
0619 14:44:27.851770 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 14:44:27.852113 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1355,"thread_start_us":150,"threads_started":1}
I20260619 14:44:27.853914 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:27.834817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48651 (local address 127.0.98.62:45229)
0619 14:44:27.838455 (+ 3638us) server_negotiation.cc:206] Beginning negotiation
0619 14:44:27.838459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:44:27.838473 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:44:27.838517 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:44:27.838520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:44:27.838630 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:44:27.838712 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:44:27.840234 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:27.841067 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:27.842763 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:27.843093 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:27.843205 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:44:27.846224 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:44:27.846228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:44:27.846232 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:44:27.846259 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:44:27.849731 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:44:27.850388 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:44:27.850391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:44:27.850393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:44:27.850478 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:44:27.850853 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:44:27.850856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:44:27.850858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:44:27.851066 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:44:27.851151 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:44:27.853631 (+ 2480us) server_negotiation.cc:299] Negotiation successful
0619 14:44:27.853755 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3555,"thread_start_us":91,"threads_started":1}
I20260619 14:44:27.855609 11863 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45229
I20260619 14:44:27.856158 11863 heartbeater.cc:464] Registering TS with master...
I20260619 14:44:27.856798 11863 heartbeater.cc:511] Master 127.0.98.62:45229 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:44:27.858608 11649 ts_manager.cc:194] Registered new tserver with Master: 7cc439f6a90c4660aa524ec20cbe37a2 (127.0.98.1:40649)
I20260619 14:44:27.860152 11649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:44:27.884923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34911/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45229
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:44:27 dist-test-slave-cr22 krb5kdc[11173](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 14:44:28.042083 11869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:44:28.042577 11869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:44:28.042701 11869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:44:28.049677 11869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:44:28.049875 11869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:44:28.050034 11869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:44:28.058647 11869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:34911/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:44:28.060354 11869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:44:28.061795 11869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:44:28.077739 11878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:44:28.077778 11875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:44:28.078236 11869 server_base.cc:1034] running on GCE node
W20260619 14:44:28.078238 11876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:44:28.078917 11869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:44:28.079720 11869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:44:28.080907 11869 hybrid_clock.cc:648] HybridClock initialized: now 1781880268080871 us; error 53 us; skew 500 ppm
Jun 19 14:44:28 dist-test-slave-cr22 krb5kdc[11173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:44:28.084583 11869 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:44:28.085701 11869 webserver.cc:460] Webserver started at http://127.0.98.2:45981/ using document root <none> and password file <none>
I20260619 14:44:28.086445 11869 fs_manager.cc:359] Metadata directory not provided
I20260619 14:44:28.086592 11869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:44:28.087002 11869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:44:28 dist-test-slave-cr22 krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:44:28.108760 11869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "457b011a137f47989b997d1705ee3b3a"
format_stamp: "Formatted at 2026-06-19 14:44:28 on dist-test-slave-cr22"
server_key: "8bbfc13966f724ef7e42fab8ba9f8e69"
server_key_iv: "20b8c0ee584002edd2a64e79bdf63493"
server_key_version: "kuduclusterkey@0"
I20260619 14:44:28.109586 11869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "457b011a137f47989b997d1705ee3b3a"
format_stamp: "Formatted at 2026-06-19 14:44:28 on dist-test-slave-cr22"
server_key: "8bbfc13966f724ef7e42fab8ba9f8e69"
server_key_iv: "20b8c0ee584002edd2a64e79bdf63493"
server_key_version: "kuduclusterkey@0"
I20260619 14:44:28.115236 11869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:44:28.137502 11885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:44:28.139163 11869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:44:28.139282 11869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "457b011a137f47989b997d1705ee3b3a"
format_stamp: "Formatted at 2026-06-19 14:44:28 on dist-test-slave-cr22"
server_key: "8bbfc13966f724ef7e42fab8ba9f8e69"
server_key_iv: "20b8c0ee584002edd2a64e79bdf63493"
server_key_version: "kuduclusterkey@0"
I20260619 14:44:28.139408 11869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:44:28.175231 11869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:44:28.179219 11869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:44:28.179526 11869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:44:28.180259 11869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:44:28.181440 11869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:44:28.181563 11869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:44:28.181674 11869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:44:28.181744 11869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:44:28.193454 11869 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33231
I20260619 14:44:28.194854 11869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:44:28.195802 11998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33231 every 8 connection(s)
I20260619 14:44:28.196746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 11869
I20260619 14:44:28.196856 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:44:28.197134 392 external_mini_cluster.cc:1428] Setting key a195eb134cdd0ec55468d09290b5a443
Jun 19 14:44:28 dist-test-slave-cr22 krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:44:28.227308 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:28.203662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40595 (local address 127.0.98.62:45229)
0619 14:44:28.203800 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:44:28.203804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:44:28.207851 (+ 4047us) server_negotiation.cc:407] Connection header received
0619 14:44:28.209228 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:44:28.209232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:44:28.209291 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:44:28.209345 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:44:28.214688 (+ 5343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.215614 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.217542 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.217888 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.217971 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:44:28.222273 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:44:28.222278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:44:28.222281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:44:28.222313 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:44:28.224231 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:44:28.224899 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:44:28.224903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:44:28.224905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:44:28.224988 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:44:28.225777 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:44:28.225781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:44:28.225782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:44:28.226008 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:44:28.226090 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:44:28.226965 (+ 875us) server_negotiation.cc:299] Negotiation successful
0619 14:44:28.227118 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:44:28.233462 12002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:28.202920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45229 (local address 127.0.98.2:40595)
0619 14:44:28.207671 (+ 4751us) negotiation.cc:107] Waiting for socket to connect
0619 14:44:28.207734 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 14:44:28.208965 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:44:28.209789 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:44:28.209799 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:44:28.210162 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:44:28.210850 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:44:28.210868 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.215802 (+ 4934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.215806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:44:28.217378 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:44:28.217391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.218194 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.218198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:44:28.218399 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:44:28.219603 (+ 1204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:44:28.219625 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:44:28.221326 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:44:28.224403 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:44:28.224409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:44:28.224428 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:44:28.224781 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:44:28.225092 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:44:28.225094 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:44:28.225096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:44:28.225267 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:44:28.226157 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:44:28.226164 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:44:28.226519 (+ 355us) client_negotiation.cc:769] Sending connection context
0619 14:44:28.227880 (+ 1361us) client_negotiation.cc:240] Negotiation successful
0619 14:44:28.230302 (+ 2422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4465,"thread_start_us":155,"threads_started":1}
I20260619 14:44:28.238286 11999 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45229
I20260619 14:44:28.238619 11999 heartbeater.cc:464] Registering TS with master...
I20260619 14:44:28.239321 11999 heartbeater.cc:511] Master 127.0.98.62:45229 requested a full tablet report, sending...
I20260619 14:44:28.240693 11649 ts_manager.cc:194] Registered new tserver with Master: 457b011a137f47989b997d1705ee3b3a (127.0.98.2:33231)
I20260619 14:44:28.241562 11649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:44:28.252760 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34911/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45229
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:44:28 dist-test-slave-cr22 krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 14:44:28.435811 12004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:44:28.436237 12004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:44:28.436342 12004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:44:28.442487 12004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:44:28.442644 12004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:44:28.443087 12004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:44:28.451149 12004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:34911/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:44:28.452868 12004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:44:28.454370 12004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:44:28.470613 12010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:44:28.470695 12009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:44:28.471295 12012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:44:28.472544 12004 server_base.cc:1034] running on GCE node
I20260619 14:44:28.473090 12004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:44:28.473919 12004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:44:28.475210 12004 hybrid_clock.cc:648] HybridClock initialized: now 1781880268475141 us; error 85 us; skew 500 ppm
Jun 19 14:44:28 dist-test-slave-cr22 krb5kdc[11173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:44:28.479775 12004 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:44:28.481192 12004 webserver.cc:460] Webserver started at http://127.0.98.3:39943/ using document root <none> and password file <none>
I20260619 14:44:28.481958 12004 fs_manager.cc:359] Metadata directory not provided
I20260619 14:44:28.482044 12004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:44:28.482266 12004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:44:28 dist-test-slave-cr22 krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:44:28.512458 12004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "962c001b7a874dc58118f26ec6f70711"
format_stamp: "Formatted at 2026-06-19 14:44:28 on dist-test-slave-cr22"
server_key: "4ad5ee34a161fa752408c1d084846461"
server_key_iv: "2e3ca6eba6779ce866521afdf2847d91"
server_key_version: "kuduclusterkey@0"
I20260619 14:44:28.521651 12004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "962c001b7a874dc58118f26ec6f70711"
format_stamp: "Formatted at 2026-06-19 14:44:28 on dist-test-slave-cr22"
server_key: "4ad5ee34a161fa752408c1d084846461"
server_key_iv: "2e3ca6eba6779ce866521afdf2847d91"
server_key_version: "kuduclusterkey@0"
I20260619 14:44:28.537241 12004 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.003s sys 0.001s
I20260619 14:44:28.567952 12019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:44:28.570040 12004 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.001s
I20260619 14:44:28.570200 12004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "962c001b7a874dc58118f26ec6f70711"
format_stamp: "Formatted at 2026-06-19 14:44:28 on dist-test-slave-cr22"
server_key: "4ad5ee34a161fa752408c1d084846461"
server_key_iv: "2e3ca6eba6779ce866521afdf2847d91"
server_key_version: "kuduclusterkey@0"
I20260619 14:44:28.570343 12004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:44:28.589718 12004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:44:28.593508 12004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:44:28.593720 12004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:44:28.594406 12004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:44:28.595466 12004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:44:28.595541 12004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:44:28.595610 12004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:44:28.595642 12004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:44:28.607025 12004 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37143
I20260619 14:44:28.608204 12004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:44:28.614856 12132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37143 every 8 connection(s)
I20260619 14:44:28.617128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 12004
I20260619 14:44:28.617241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:44:28.617527 392 external_mini_cluster.cc:1428] Setting key 60ffc41e8b4bd05f0e22ebfaaeae4e4b
Jun 19 14:44:28 dist-test-slave-cr22 krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:44:28.632614 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:28.618897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41869 (local address 127.0.98.62:45229)
0619 14:44:28.619537 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0619 14:44:28.619541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:44:28.619556 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:44:28.620501 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:44:28.620505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:44:28.620554 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:44:28.620604 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:44:28.622263 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.623142 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.624860 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.625199 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.625270 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:44:28.628331 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:44:28.628335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:44:28.628338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:44:28.628367 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:44:28.630102 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:44:28.630831 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:44:28.630835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:44:28.630837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:44:28.630920 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:44:28.631296 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:44:28.631299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:44:28.631301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:44:28.631552 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:44:28.631632 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:44:28.632317 (+ 685us) server_negotiation.cc:299] Negotiation successful
0619 14:44:28.632439 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566}
I20260619 14:44:28.634235 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:28.617783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45229 (local address 127.0.98.3:41869)
0619 14:44:28.619198 (+ 1415us) negotiation.cc:107] Waiting for socket to connect
0619 14:44:28.619254 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:44:28.620253 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:44:28.621003 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:44:28.621012 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:44:28.621418 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:44:28.622106 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:44:28.622123 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.623337 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.623340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:44:28.624723 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:44:28.624734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.625465 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.625468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:44:28.625673 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:44:28.626374 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:44:28.626394 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:44:28.628087 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:44:28.630246 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:44:28.630252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:44:28.630271 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:44:28.630716 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:44:28.631020 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:44:28.631023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:44:28.631025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:44:28.631195 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:44:28.631697 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:44:28.631704 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:44:28.632044 (+ 340us) client_negotiation.cc:769] Sending connection context
0619 14:44:28.632837 (+ 793us) client_negotiation.cc:240] Negotiation successful
0619 14:44:28.633187 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1185,"thread_start_us":175,"threads_started":1}
I20260619 14:44:28.635550 12133 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45229
I20260619 14:44:28.635807 12133 heartbeater.cc:464] Registering TS with master...
I20260619 14:44:28.636381 12133 heartbeater.cc:511] Master 127.0.98.62:45229 requested a full tablet report, sending...
I20260619 14:44:28.637598 11649 ts_manager.cc:194] Registered new tserver with Master: 962c001b7a874dc58118f26ec6f70711 (127.0.98.3:37143)
I20260619 14:44:28.638314 11649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41869
I20260619 14:44:28.643196 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:44:28.655071 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:28.645409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49564 (local address 127.0.98.62:45229)
0619 14:44:28.645567 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 14:44:28.645572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:44:28.645961 (+ 389us) server_negotiation.cc:407] Connection header received
0619 14:44:28.646091 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:44:28.646094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:44:28.646140 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:44:28.646219 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:44:28.647116 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.647991 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.649329 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.649676 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.649774 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:44:28.651535 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:44:28.651539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:44:28.651542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:44:28.651568 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:44:28.653260 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:44:28.653786 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:44:28.653790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:44:28.653792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:44:28.653876 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:44:28.654173 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:44:28.654176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:44:28.654178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:44:28.654383 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:44:28.654518 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 14:44:28.654786 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 14:44:28.654905 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:44:28.658555 11649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:44:28.662969 11649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:44:28.681862 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:28.673824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37143 (local address 127.0.0.1:45208)
0619 14:44:28.674678 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0619 14:44:28.674776 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0619 14:44:28.674901 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:44:28.675733 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:44:28.675740 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:44:28.675764 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:44:28.675980 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:44:28.675987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.677706 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.677710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:44:28.679247 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:44:28.679257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.681325 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.681328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:44:28.681442 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:44:28.681478 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:44:28.681597 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 14:44:28.681711 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":115,"threads_started":1}
I20260619 14:44:28.683561 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:28.674569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45208 (local address 127.0.98.3:37143)
0619 14:44:28.675166 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0619 14:44:28.675176 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:44:28.675198 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:44:28.675268 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:44:28.675273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:44:28.675521 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:44:28.676134 (+ 613us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:44:28.676255 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.677579 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.680200 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.681016 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.681121 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:44:28.681266 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:44:28.683246 (+ 1980us) server_negotiation.cc:299] Negotiation successful
0619 14:44:28.683361 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":201,"threads_started":1}
I20260619 14:44:28.687108 12064 tablet_service.cc:1467] Processing CreateTablet for tablet 4e1ec33ab46f45e8bf1f1135e86ef2b1 (DEFAULT_TABLE table=test-table [id=581c44b45a8b4778bb4771496cd38567]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:44:28.687801 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:28.676058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33231 (local address 127.0.0.1:34966)
0619 14:44:28.678559 (+ 2501us) negotiation.cc:107] Waiting for socket to connect
0619 14:44:28.678574 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:44:28.678677 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:44:28.680625 (+ 1948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:44:28.680627 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:44:28.680644 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:44:28.680746 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:44:28.680753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.682500 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.682504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:44:28.683730 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:44:28.683741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.687292 (+ 3551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.687296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:44:28.687409 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:44:28.687462 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 14:44:28.687554 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 14:44:28.687649 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2436,"thread_start_us":99,"threads_started":1}
I20260619 14:44:28.688918 12064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e1ec33ab46f45e8bf1f1135e86ef2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:44:28.688997 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:28.676510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34966 (local address 127.0.98.2:33231)
0619 14:44:28.678016 (+ 1506us) server_negotiation.cc:206] Beginning negotiation
0619 14:44:28.678031 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:44:28.678802 (+ 771us) server_negotiation.cc:407] Connection header received
0619 14:44:28.678909 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:44:28.678915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:44:28.679141 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:44:28.679308 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:44:28.680886 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.682358 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.684712 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.685580 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.685673 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:44:28.685834 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 14:44:28.688055 (+ 2221us) server_negotiation.cc:299] Negotiation successful
0619 14:44:28.688174 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1285,"thread_start_us":99,"threads_started":1}
I20260619 14:44:28.692350 11933 tablet_service.cc:1467] Processing CreateTablet for tablet 4e1ec33ab46f45e8bf1f1135e86ef2b1 (DEFAULT_TABLE table=test-table [id=581c44b45a8b4778bb4771496cd38567]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:44:28.693586 11933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e1ec33ab46f45e8bf1f1135e86ef2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:44:28.698763 12151 tablet_bootstrap.cc:492] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711: Bootstrap starting.
I20260619 14:44:28.701695 12151 tablet_bootstrap.cc:654] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711: Neither blocks nor log segments found. Creating new log.
I20260619 14:44:28.702630 12151 log.cc:824] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711: Log is configured to *not* fsync() on all Append() calls
I20260619 14:44:28.704782 12151 tablet_bootstrap.cc:492] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711: No bootstrap required, opened a new log
I20260619 14:44:28.704968 12151 ts_tablet_manager.cc:1397] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:44:28.705554 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:28.673981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40649 (local address 127.0.0.1:54816)
0619 14:44:28.678180 (+ 4199us) negotiation.cc:107] Waiting for socket to connect
0619 14:44:28.678197 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:44:28.678292 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:44:28.680358 (+ 2066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:44:28.680361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:44:28.680377 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:44:28.680476 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:44:28.680483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.701875 (+ 21392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.701880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:44:28.703660 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:44:28.703674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.705103 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.705107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:44:28.705207 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:44:28.705233 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:44:28.705310 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 14:44:28.705396 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4130,"thread_start_us":96,"threads_started":1}
I20260619 14:44:28.709475 12151 raft_consensus.cc:348] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962c001b7a874dc58118f26ec6f70711" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37143 } } peers { permanent_uuid: "7cc439f6a90c4660aa524ec20cbe37a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40649 } } peers { permanent_uuid: "457b011a137f47989b997d1705ee3b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33231 } }
I20260619 14:44:28.709853 12151 raft_consensus.cc:374] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:44:28.709940 12151 raft_consensus.cc:729] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 962c001b7a874dc58118f26ec6f70711, State: Initialized, Role: FOLLOWER
I20260619 14:44:28.710469 12151 consensus_queue.cc:260] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962c001b7a874dc58118f26ec6f70711" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37143 } } peers { permanent_uuid: "7cc439f6a90c4660aa524ec20cbe37a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40649 } } peers { permanent_uuid: "457b011a137f47989b997d1705ee3b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33231 } }
I20260619 14:44:28.711578 12151 ts_tablet_manager.cc:1428] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20260619 14:44:28.711652 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:28.675175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54816 (local address 127.0.98.1:40649)
0619 14:44:28.679648 (+ 4473us) server_negotiation.cc:206] Beginning negotiation
0619 14:44:28.679655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:44:28.679681 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:44:28.679881 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:44:28.679887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:44:28.680077 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:44:28.680219 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:44:28.685921 (+ 5702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.687107 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.703858 (+ 16751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.704733 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.704849 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:44:28.705017 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 14:44:28.709967 (+ 4950us) server_negotiation.cc:299] Negotiation successful
0619 14:44:28.711486 (+ 1519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4356,"thread_start_us":114,"threads_started":1}
I20260619 14:44:28.712181 12133 heartbeater.cc:503] Master 127.0.98.62:45229 was elected leader, sending a full tablet report...
I20260619 14:44:28.714843 11797 tablet_service.cc:1467] Processing CreateTablet for tablet 4e1ec33ab46f45e8bf1f1135e86ef2b1 (DEFAULT_TABLE table=test-table [id=581c44b45a8b4778bb4771496cd38567]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:44:28.716048 11797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e1ec33ab46f45e8bf1f1135e86ef2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:44:28.720803 12153 tablet_bootstrap.cc:492] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a: Bootstrap starting.
I20260619 14:44:28.723321 12153 tablet_bootstrap.cc:654] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a: Neither blocks nor log segments found. Creating new log.
I20260619 14:44:28.724835 12153 log.cc:824] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:44:28.725751 12155 tablet_bootstrap.cc:492] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2: Bootstrap starting.
I20260619 14:44:28.728426 12155 tablet_bootstrap.cc:654] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2: Neither blocks nor log segments found. Creating new log.
I20260619 14:44:28.728451 12153 tablet_bootstrap.cc:492] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a: No bootstrap required, opened a new log
I20260619 14:44:28.728636 12153 ts_tablet_manager.cc:1397] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 14:44:28.729410 12155 log.cc:824] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2: Log is configured to *not* fsync() on all Append() calls
I20260619 14:44:28.731688 12155 tablet_bootstrap.cc:492] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2: No bootstrap required, opened a new log
I20260619 14:44:28.731899 12155 ts_tablet_manager.cc:1397] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:44:28.732798 12153 raft_consensus.cc:348] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962c001b7a874dc58118f26ec6f70711" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37143 } } peers { permanent_uuid: "7cc439f6a90c4660aa524ec20cbe37a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40649 } } peers { permanent_uuid: "457b011a137f47989b997d1705ee3b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33231 } }
I20260619 14:44:28.733074 12153 raft_consensus.cc:374] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:44:28.733122 12153 raft_consensus.cc:729] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 457b011a137f47989b997d1705ee3b3a, State: Initialized, Role: FOLLOWER
I20260619 14:44:28.734102 12153 consensus_queue.cc:260] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962c001b7a874dc58118f26ec6f70711" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37143 } } peers { permanent_uuid: "7cc439f6a90c4660aa524ec20cbe37a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40649 } } peers { permanent_uuid: "457b011a137f47989b997d1705ee3b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33231 } }
I20260619 14:44:28.735175 12153 ts_tablet_manager.cc:1428] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:44:28.735607 11999 heartbeater.cc:503] Master 127.0.98.62:45229 was elected leader, sending a full tablet report...
I20260619 14:44:28.736901 12155 raft_consensus.cc:348] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962c001b7a874dc58118f26ec6f70711" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37143 } } peers { permanent_uuid: "7cc439f6a90c4660aa524ec20cbe37a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40649 } } peers { permanent_uuid: "457b011a137f47989b997d1705ee3b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33231 } }
I20260619 14:44:28.737270 12155 raft_consensus.cc:374] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:44:28.737366 12155 raft_consensus.cc:729] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cc439f6a90c4660aa524ec20cbe37a2, State: Initialized, Role: FOLLOWER
I20260619 14:44:28.737939 12155 consensus_queue.cc:260] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962c001b7a874dc58118f26ec6f70711" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37143 } } peers { permanent_uuid: "7cc439f6a90c4660aa524ec20cbe37a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40649 } } peers { permanent_uuid: "457b011a137f47989b997d1705ee3b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33231 } }
I20260619 14:44:28.739089 12155 ts_tablet_manager.cc:1428] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260619 14:44:28.739573 11863 heartbeater.cc:503] Master 127.0.98.62:45229 was elected leader, sending a full tablet report...
I20260619 14:44:28.766166 12159 raft_consensus.cc:482] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:44:28.766427 12159 raft_consensus.cc:504] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962c001b7a874dc58118f26ec6f70711" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37143 } } peers { permanent_uuid: "7cc439f6a90c4660aa524ec20cbe37a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40649 } } peers { permanent_uuid: "457b011a137f47989b997d1705ee3b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33231 } }
I20260619 14:44:28.768523 12159 leader_election.cc:288] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 962c001b7a874dc58118f26ec6f70711 (127.0.98.3:37143), 457b011a137f47989b997d1705ee3b3a (127.0.98.2:33231)
I20260619 14:44:28.773629 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:28.768410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39455 (local address 127.0.98.3:37143)
0619 14:44:28.768538 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:44:28.768544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:44:28.768945 (+ 401us) server_negotiation.cc:407] Connection header received
0619 14:44:28.769164 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:44:28.769168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:44:28.769218 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:44:28.769270 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:44:28.769762 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.770535 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.772252 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.772873 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.772958 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:44:28.773030 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:44:28.773389 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 14:44:28.773473 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:44:28.774377 12160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:28.767985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37143 (local address 127.0.98.1:39455)
0619 14:44:28.768853 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0619 14:44:28.768868 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:44:28.769027 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:44:28.769422 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:44:28.769425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:44:28.769444 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:44:28.769586 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:44:28.769629 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.770722 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.770725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:44:28.772101 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:44:28.772113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.773131 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.773134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:44:28.773235 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:44:28.773261 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:44:28.774012 (+ 751us) client_negotiation.cc:240] Negotiation successful
0619 14:44:28.774216 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":131,"threads_started":1}
I20260619 14:44:28.775362 12087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e1ec33ab46f45e8bf1f1135e86ef2b1" candidate_uuid: "7cc439f6a90c4660aa524ec20cbe37a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962c001b7a874dc58118f26ec6f70711" is_pre_election: true
I20260619 14:44:28.775807 12087 raft_consensus.cc:2454] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cc439f6a90c4660aa524ec20cbe37a2 in term 0.
I20260619 14:44:28.776532 11749 leader_election.cc:302] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cc439f6a90c4660aa524ec20cbe37a2, 962c001b7a874dc58118f26ec6f70711; no voters:
I20260619 14:44:28.776949 12159 raft_consensus.cc:2781] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:44:28.777048 12159 raft_consensus.cc:482] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:44:28.777097 12159 raft_consensus.cc:3037] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:44:28.778932 12159 raft_consensus.cc:504] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962c001b7a874dc58118f26ec6f70711" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37143 } } peers { permanent_uuid: "7cc439f6a90c4660aa524ec20cbe37a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40649 } } peers { permanent_uuid: "457b011a137f47989b997d1705ee3b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33231 } }
I20260619 14:44:28.779337 12159 leader_election.cc:288] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 [CANDIDATE]: Term 1 election: Requested vote from peers 962c001b7a874dc58118f26ec6f70711 (127.0.98.3:37143), 457b011a137f47989b997d1705ee3b3a (127.0.98.2:33231)
I20260619 14:44:28.780143 12087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e1ec33ab46f45e8bf1f1135e86ef2b1" candidate_uuid: "7cc439f6a90c4660aa524ec20cbe37a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962c001b7a874dc58118f26ec6f70711"
I20260619 14:44:28.780285 12087 raft_consensus.cc:3037] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:44:28.781698 12087 raft_consensus.cc:2454] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cc439f6a90c4660aa524ec20cbe37a2 in term 1.
I20260619 14:44:28.782050 11749 leader_election.cc:302] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cc439f6a90c4660aa524ec20cbe37a2, 962c001b7a874dc58118f26ec6f70711; no voters:
I20260619 14:44:28.782567 12161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:28.767998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33231 (local address 127.0.98.1:47249)
0619 14:44:28.774972 (+ 6974us) negotiation.cc:107] Waiting for socket to connect
0619 14:44:28.774987 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:44:28.775163 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:44:28.775566 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:44:28.775569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:44:28.775586 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:44:28.775755 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:44:28.775762 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.777079 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.777095 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:44:28.779935 (+ 2840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:44:28.779944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.782111 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.782118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:44:28.782223 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:44:28.782247 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:44:28.782309 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 14:44:28.782388 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6693,"mutex_wait_us":16,"thread_start_us":100,"threads_started":1}
I20260619 14:44:28.782960 12159 raft_consensus.cc:2781] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:44:28.783192 12159 raft_consensus.cc:686] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 [term 1 LEADER]: Becoming Leader. State: Replica: 7cc439f6a90c4660aa524ec20cbe37a2, State: Running, Role: LEADER
I20260619 14:44:28.783638 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:28.768462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47249 (local address 127.0.98.2:33231)
0619 14:44:28.768664 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 14:44:28.768669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:44:28.775064 (+ 6395us) server_negotiation.cc:407] Connection header received
0619 14:44:28.775291 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:44:28.775294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:44:28.775342 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:44:28.775395 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:44:28.775895 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.776656 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.781160 (+ 4504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.781854 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.781963 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:44:28.782035 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:44:28.783410 (+ 1375us) server_negotiation.cc:299] Negotiation successful
0619 14:44:28.783521 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260619 14:44:28.784308 11938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e1ec33ab46f45e8bf1f1135e86ef2b1" candidate_uuid: "7cc439f6a90c4660aa524ec20cbe37a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "457b011a137f47989b997d1705ee3b3a" is_pre_election: true
I20260619 14:44:28.784607 11938 raft_consensus.cc:2454] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cc439f6a90c4660aa524ec20cbe37a2 in term 0.
I20260619 14:44:28.785379 11943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e1ec33ab46f45e8bf1f1135e86ef2b1" candidate_uuid: "7cc439f6a90c4660aa524ec20cbe37a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "457b011a137f47989b997d1705ee3b3a"
I20260619 14:44:28.785490 11943 raft_consensus.cc:3037] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:44:28.786518 11943 raft_consensus.cc:2454] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cc439f6a90c4660aa524ec20cbe37a2 in term 1.
I20260619 14:44:28.783612 12159 consensus_queue.cc:237] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962c001b7a874dc58118f26ec6f70711" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37143 } } peers { permanent_uuid: "7cc439f6a90c4660aa524ec20cbe37a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40649 } } peers { permanent_uuid: "457b011a137f47989b997d1705ee3b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33231 } }
I20260619 14:44:28.790895 11649 catalog_manager.cc:5495] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cc439f6a90c4660aa524ec20cbe37a2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7cc439f6a90c4660aa524ec20cbe37a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962c001b7a874dc58118f26ec6f70711" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cc439f6a90c4660aa524ec20cbe37a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "457b011a137f47989b997d1705ee3b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33231 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:44:28 dist-test-slave-cr22 krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:44:28.818949 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:28.814200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54824 (local address 127.0.98.1:40649)
0619 14:44:28.814319 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:44:28.814324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:44:28.814337 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:44:28.814428 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:44:28.814431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:44:28.814501 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:44:28.814614 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:44:28.815007 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.815833 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.816947 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:44:28.817230 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:44:28.817345 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:44:28.817701 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:44:28.818312 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:44:28.818450 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 14:44:28.818692 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 14:44:28.818770 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:44:28.828470 11943 raft_consensus.cc:1261] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a [term 1 FOLLOWER]: Refusing update from remote peer 7cc439f6a90c4660aa524ec20cbe37a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:44:28.829284 12159 consensus_queue.cc:1035] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "457b011a137f47989b997d1705ee3b3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:44:28.836274 12087 raft_consensus.cc:1261] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711 [term 1 FOLLOWER]: Refusing update from remote peer 7cc439f6a90c4660aa524ec20cbe37a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:44:28.836961 12159 consensus_queue.cc:1035] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "962c001b7a874dc58118f26ec6f70711" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:44:28.837357 11864 tablet.cc:2369] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 7cc439f6a90c4660aa524ec20cbe37a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:44:28.852401 12166 mvcc.cc:205] Tried to move back new op lower bound from 7298581581111566336 to 7298581580955398144. Current Snapshot: MvccSnapshot[applied={T|T < 7298581581111566336}]
I20260619 14:44:28.861586 12169 mvcc.cc:205] Tried to move back new op lower bound from 7298581581111566336 to 7298581580955398144. Current Snapshot: MvccSnapshot[applied={T|T < 7298581581111566336}]
I20260619 14:44:28.863164 12167 mvcc.cc:205] Tried to move back new op lower bound from 7298581581111566336 to 7298581580955398144. Current Snapshot: MvccSnapshot[applied={T|T < 7298581581111566336}]
I20260619 14:44:28.874801 11649 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:44:28.875000 11649 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:44:28.878304 11649 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed388cdcc60f492bab90574ec9e2ec47: Sending DeleteTablet for 3 replicas of tablet 4e1ec33ab46f45e8bf1f1135e86ef2b1
I20260619 14:44:28.879580 12064 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e1ec33ab46f45e8bf1f1135e86ef2b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:44:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45208
I20260619 14:44:28.880155 11797 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e1ec33ab46f45e8bf1f1135e86ef2b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:44:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54816
I20260619 14:44:28.881062 11933 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e1ec33ab46f45e8bf1f1135e86ef2b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:44:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34966
I20260619 14:44:28.882591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 11730
I20260619 14:44:28.887497 12178 tablet_replica.cc:330] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a: stopping tablet replica
I20260619 14:44:28.888172 12178 raft_consensus.cc:2229] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:44:28.894179 12178 raft_consensus.cc:2258] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:44:28.894779 12176 tablet_replica.cc:330] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711: stopping tablet replica
I20260619 14:44:28.896477 12178 ts_tablet_manager.cc:1905] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:44:28.899617 12178 ts_tablet_manager.cc:1918] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:44:28.899753 12178 log.cc:1196] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/4e1ec33ab46f45e8bf1f1135e86ef2b1
I20260619 14:44:28.900058 12178 ts_tablet_manager.cc:1939] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 457b011a137f47989b997d1705ee3b3a: Deleting consensus metadata
I20260619 14:44:28.901557 11636 catalog_manager.cc:4841] TS 457b011a137f47989b997d1705ee3b3a (127.0.98.2:33231): tablet 4e1ec33ab46f45e8bf1f1135e86ef2b1 (table test-table [id=581c44b45a8b4778bb4771496cd38567]) successfully deleted
I20260619 14:44:28.901723 12176 raft_consensus.cc:2229] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:44:28.902087 12176 raft_consensus.cc:2258] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:44:28.904064 12176 ts_tablet_manager.cc:1905] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:44:28.907094 12176 ts_tablet_manager.cc:1918] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:44:28.907204 12176 log.cc:1196] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/4e1ec33ab46f45e8bf1f1135e86ef2b1
I20260619 14:44:28.907599 12176 ts_tablet_manager.cc:1939] T 4e1ec33ab46f45e8bf1f1135e86ef2b1 P 962c001b7a874dc58118f26ec6f70711: Deleting consensus metadata
I20260619 14:44:28.909030 11634 catalog_manager.cc:4841] TS 962c001b7a874dc58118f26ec6f70711 (127.0.98.3:37143): tablet 4e1ec33ab46f45e8bf1f1135e86ef2b1 (table test-table [id=581c44b45a8b4778bb4771496cd38567]) successfully deleted
W20260619 14:44:28.909349 11635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40649 (error 108)
I20260619 14:44:28.909997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 11869
I20260619 14:44:28.911278 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:44:28.910232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40649 (local address 127.0.0.1:54838)
0619 14:44:28.911139 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0619 14:44:28.911197 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":828}
W20260619 14:44:28.911422 11635 catalog_manager.cc:4584] TS 7cc439f6a90c4660aa524ec20cbe37a2 (127.0.98.1:40649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e1ec33ab46f45e8bf1f1135e86ef2b1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40649: connect: Connection refused (error 111)
I20260619 14:44:28.922785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 12004
I20260619 14:44:28.930008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 11617
I20260619 14:44:28.939165 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 14:44:29.274524 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 14:44:29.274650 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 14:44:29.620442 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 14:44:29.620 UTC [11203] LOG: received smart shutdown request
2026-06-19 14:44:29.624 UTC [11203] LOG: background worker "logical replication launcher" (PID 11212) exited with exit code 1
2026-06-19 14:44:29.625 UTC [11207] LOG: shutting down
2026-06-19 14:44:29.837 UTC [11203] LOG: database system is shut down
2026-06-19T14:44:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49739 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:44:30 dist-test-slave-cr22 krb5kdc[12188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:44:30 dist-test-slave-cr22 krb5kdc[12188](info): set up 2 sockets
Jun 19 14:44:30 dist-test-slave-cr22 krb5kdc[12188](info): commencing operation
krb5kdc: starting...
W20260619 14:44:31.015623 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:44:31 dist-test-slave-cr22 krb5kdc[12188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:44:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:44:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 14:44:31.212330 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZ9bYsM/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/postgres -l logfile start
2026-06-19 14:44:33.915 UTC [12218] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 14:44:33.915 UTC [12218] LOG: listening on IPv4 address "127.0.98.20", port 48437
2026-06-19 14:44:33.919 UTC [12218] LOG: listening on Unix socket "/tmp/.s.PGSQL.48437"
2026-06-19 14:44:33.942 UTC [12221] LOG: database system was shut down at 2026-06-19 14:44:32 UTC
2026-06-19 14:44:33.946 UTC [12218] LOG: database system is ready to accept connections
I20260619 14:44:34.911296 392 mini_postgres.cc:96] Postgres bound to 48437
2026-06-19 14:44:34.917 UTC [12229] FATAL: database "slave" does not exist
127.0.98.20:48437 - accepting connections
I20260619 14:44:34.918262 392 mini_ranger.cc:161] Starting Ranger...
I20260619 14:44:34.933830 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 14:44:35.488451 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 14:44:35.488546 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ranger-admin
2026-06-19 14:44:35,774 [I] DB FLAVOR :POSTGRES
2026-06-19 14:44:35,774 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:44:35,774 [I] Checking connection..
2026-06-19 14:44:35,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 14:44:36,026 [I] Checking connection passed.
2026-06-19 14:44:36,027 [I] --------- Verifying version history table ---------
2026-06-19 14:44:36,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:44:36,286 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:44:36,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:44:36,548 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:44:36,548 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 14:44:36,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 14:44:36.814 UTC [12293] WARNING: there is no transaction in progress
2026-06-19 14:44:36,821 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 14:44:36,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:44:37,106 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 14:44:37,106 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:44:37,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 14:44:37,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 14:44:37,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cr22.c.gcp-upstream.internal','N') ;"
2026-06-19 14:44:37,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:44:38,140 [I] Table x_portal_user does not exist in database ranger
2026-06-19 14:44:38,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:44:38,424 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 14:44:38,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:44:38,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:44:38,914 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 14:44:38,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 14:44:39.986 UTC [12401] WARNING: there is no transaction in progress
2026-06-19 14:44:40.004 UTC [12401] WARNING: there is no transaction in progress
2026-06-19 14:44:40.024 UTC [12401] WARNING: there is no transaction in progress
2026-06-19 14:44:40.042 UTC [12401] WARNING: there is no transaction in progress
2026-06-19 14:44:40.063 UTC [12401] WARNING: there is no transaction in progress
2026-06-19 14:44:40.085 UTC [12401] WARNING: there is no transaction in progress
2026-06-19 14:44:40.150 UTC [12401] WARNING: there is no transaction in progress
2026-06-19 14:44:40.158 UTC [12401] WARNING: there is no transaction in progress
2026-06-19 14:44:40.174 UTC [12401] WARNING: there is no transaction in progress
2026-06-19 14:44:40.186 UTC [12401] WARNING: there is no transaction in progress
2026-06-19 14:44:40.198 UTC [12401] WARNING: there is no transaction in progress
2026-06-19 14:44:40.216 UTC [12401] WARNING: there is no transaction in progress
2026-06-19 14:44:40.228 UTC [12401] WARNING: there is no transaction in progress
2026-06-19 14:44:40.236 UTC [12401] WARNING: there is no transaction in progress
2026-06-19 14:44:40.246 UTC [12401] WARNING: there is no transaction in progress
2026-06-19 14:44:40.255 UTC [12401] WARNING: there is no transaction in progress
2026-06-19 14:44:41,125 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 14:44:41,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 14:44:41,377 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 14:44:41,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:44:41,655 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 14:44:41,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:44:41,937 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 14:44:41,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:44:42,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:44:42,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cr22.c.gcp-upstream.internal';"
2026-06-19 14:44:42,718 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 14:44:42,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 14:44:42,988 [I] DB_PATCHES have already been applied
I20260619 14:44:42.995085 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZ9bYsM/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 14:44:42.995187 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 14:44:42.998765 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:36857
Jun 19, 2026 2:44:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:44:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:44:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:44:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:44:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:44:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:44:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 14:44:44.148838 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.231s user 0.000s sys 0.010s
Jun 19, 2026 2:45:04 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:45:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 2:45:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:45:04 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:45:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:45:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:45:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:45:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:45:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:45:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:45:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:45:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:45:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:45:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:45:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:45:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:45:04 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:45:05 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 14:45:05.691823 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 14:45:05 dist-test-slave-cr22 krb5kdc[12188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880305, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 14:45:05.794056 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 14:45:05.804458 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 14:45:06.830317 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 14:45:06.830410 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ranger-kms
I20260619 14:45:06.830433 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:48437
2026-06-19 14:45:07,001 [I] DB FLAVOR :POSTGRES
2026-06-19 14:45:07,001 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:45:07,001 [I] Checking connection
2026-06-19 14:45:07,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 14:45:07,234 [I] connection success
2026-06-19 14:45:07,234 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 14:45:07,234 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 14:45:07,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 14:45:07,470 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 14:45:07,470 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:45:07,470 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 14:45:07,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48437/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 14:45:07,756 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 14:45:08.249222 392 mini_ranger_kms.cc:310] Created kms service
I20260619 14:45:08.483661 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 14:45:08.539259 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 14:45:08 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880305, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:45:08.978870 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 14:45:08.979053 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZ9bYsM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZ9bYsM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 14:45:08.979079 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 14:45:08.982928 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12581
I20260619 14:45:08.983022 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:43053
Jun 19, 2026 2:45:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:45:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:45:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:45:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:45:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:45:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 14:45:11.296993 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.503s user 0.006s sys 0.149s
I20260619 14:45:11.297120 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 14:45:11.297187 392 mini_ranger_kms.cc:401] 127.0.98.20:43053/kms/v1/keys
Jun 19 14:45:11 dist-test-slave-cr22 krb5kdc[12188](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880311, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 14:45:11 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880311, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 14:45:12 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880311, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 14:45:14 dist-test-slave-cr22 krb5kdc[12188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:45:14.556543 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45175
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45175
--ranger_config_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43053/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:45:14.672659 12632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:14.672964 12632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:14.673035 12632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:14.678638 12632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:45:14.678730 12632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:14.678766 12632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:14.678802 12632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:45:14.678836 12632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:45:14.684840 12632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:43053/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45175
--ranger_config_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:14.686017 12632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:14.687055 12632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:14.696460 12638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:14.696519 12637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:14.696655 12632 server_base.cc:1034] running on GCE node
W20260619 14:45:14.696834 12640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:14.697532 12632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:14.698917 12632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:14.700115 12632 hybrid_clock.cc:648] HybridClock initialized: now 1781880314700076 us; error 46 us; skew 500 ppm
Jun 19 14:45:14 dist-test-slave-cr22 krb5kdc[12188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:14.703896 12632 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:45:14.704964 12632 webserver.cc:460] Webserver started at http://127.0.98.62:36361/ using document root <none> and password file <none>
I20260619 14:45:14.705523 12632 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:14.705579 12632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:14.705809 12632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:45:14 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:45:14.757719 12632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "08ab7d33e6804e54b64770e789b36ca6"
format_stamp: "Formatted at 2026-06-19 14:45:14 on dist-test-slave-cr22"
server_key: "a2d3e8a8f0bce433634ddd8cfe940d41"
server_key_iv: "a3a9b5d229ac0946bc1e61c4a8a41cad"
server_key_version: "kuduclusterkey@0"
I20260619 14:45:14.758373 12632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "08ab7d33e6804e54b64770e789b36ca6"
format_stamp: "Formatted at 2026-06-19 14:45:14 on dist-test-slave-cr22"
server_key: "a2d3e8a8f0bce433634ddd8cfe940d41"
server_key_iv: "a3a9b5d229ac0946bc1e61c4a8a41cad"
server_key_version: "kuduclusterkey@0"
I20260619 14:45:14.762507 12632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:45:14.781183 12647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:14.782697 12632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:45:14.782857 12632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "08ab7d33e6804e54b64770e789b36ca6"
format_stamp: "Formatted at 2026-06-19 14:45:14 on dist-test-slave-cr22"
server_key: "a2d3e8a8f0bce433634ddd8cfe940d41"
server_key_iv: "a3a9b5d229ac0946bc1e61c4a8a41cad"
server_key_version: "kuduclusterkey@0"
I20260619 14:45:14.782995 12632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:14.797544 12632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:14.800920 12632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:14.801146 12632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:14.809345 12632 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45175
I20260619 14:45:14.809376 12699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45175 every 8 connection(s)
I20260619 14:45:14.810338 12632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:45:14.813261 12700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:14.814970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 12632
I20260619 14:45:14.815092 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:45:14.815379 392 external_mini_cluster.cc:1428] Setting key 88f9c282da96ce194967f7a6d4be276b
I20260619 14:45:14.820052 12700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6: Bootstrap starting.
I20260619 14:45:14.822485 12700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6: Neither blocks nor log segments found. Creating new log.
I20260619 14:45:14.823233 12700 log.cc:824] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:45:14 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:45:14.825510 12700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6: No bootstrap required, opened a new log
I20260619 14:45:14.829620 12700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ab7d33e6804e54b64770e789b36ca6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } }
I20260619 14:45:14.829959 12700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:45:14.830041 12700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ab7d33e6804e54b64770e789b36ca6, State: Initialized, Role: FOLLOWER
I20260619 14:45:14.830595 12700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ab7d33e6804e54b64770e789b36ca6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } }
I20260619 14:45:14.830754 12700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:45:14.830843 12700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:45:14.830945 12700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:45:14.831650 12703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:14.816864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53118 (local address 127.0.98.62:45175)
0619 14:45:14.817329 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:14.817350 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:14.817391 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:45:14.818284 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:14.818308 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:14.818747 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:14.819109 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:14.819939 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:14.821506 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:14.822675 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:14.823355 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:14.823537 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:14.826144 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:14.826150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:14.826163 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:14.826189 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:14.828473 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:14.828923 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:14.828929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:14.828935 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:14.829043 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:14.829328 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:14.829331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:14.829333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:14.829737 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:14.830103 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:14.830255 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 14:45:14.830697 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":139,"threads_started":1}
I20260619 14:45:14.832126 12700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ab7d33e6804e54b64770e789b36ca6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } }
I20260619 14:45:14.832499 12700 leader_election.cc:302] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08ab7d33e6804e54b64770e789b36ca6; no voters:
I20260619 14:45:14.832911 12700 leader_election.cc:288] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:45:14.833015 12705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:45:14.833210 12705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 08ab7d33e6804e54b64770e789b36ca6, State: Running, Role: LEADER
I20260619 14:45:14.833494 12705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ab7d33e6804e54b64770e789b36ca6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } }
I20260619 14:45:14.834185 12700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:45:14.836256 12706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08ab7d33e6804e54b64770e789b36ca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ab7d33e6804e54b64770e789b36ca6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } } }
I20260619 14:45:14.836622 12706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:45:14.836638 12719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08ab7d33e6804e54b64770e789b36ca6. Latest consensus state: current_term: 1 leader_uuid: "08ab7d33e6804e54b64770e789b36ca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ab7d33e6804e54b64770e789b36ca6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } } }
I20260619 14:45:14.836830 12719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:45:14.836472 12700 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:45:14.840479 12720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:45:14.845184 12720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:45:14.852294 12720 catalog_manager.cc:1347] Generated new cluster ID: f44b40a8eaf944a8bce34b9d33b2a79b
I20260619 14:45:14.852370 12720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:45:14.863101 12720 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:45:14.864621 12720 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:45:14.880584 12720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6: Generated new TSK 0
I20260619 14:45:14.881731 12720 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:45:15.583945 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43053/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45175
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:45:15.706720 12746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:15.706985 12746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:15.707036 12746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:15.711357 12746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:15.711417 12746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:15.711566 12746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:45:15.716971 12746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:43053/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:15.718163 12746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:15.719175 12746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:15.731334 12751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:15.731571 12754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:15.731343 12752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:15.732012 12746 server_base.cc:1034] running on GCE node
I20260619 14:45:15.732545 12746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:15.733453 12746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:15.734680 12746 hybrid_clock.cc:648] HybridClock initialized: now 1781880315734657 us; error 38 us; skew 500 ppm
Jun 19 14:45:15 dist-test-slave-cr22 krb5kdc[12188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:15.738574 12746 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:45:15.739825 12746 webserver.cc:460] Webserver started at http://127.0.98.1:38135/ using document root <none> and password file <none>
I20260619 14:45:15.740624 12746 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:15.740926 12746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:15.741195 12746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:45:15 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:45:15.758788 12746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "ac86cab385c444e198db095959aba51b"
format_stamp: "Formatted at 2026-06-19 14:45:15 on dist-test-slave-cr22"
server_key: "2a358950f56756cd4ac4d69a567c3286"
server_key_iv: "6475b26e46b6880bafc499ba526374c6"
server_key_version: "kuduclusterkey@0"
I20260619 14:45:15.759559 12746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ac86cab385c444e198db095959aba51b"
format_stamp: "Formatted at 2026-06-19 14:45:15 on dist-test-slave-cr22"
server_key: "2a358950f56756cd4ac4d69a567c3286"
server_key_iv: "6475b26e46b6880bafc499ba526374c6"
server_key_version: "kuduclusterkey@0"
I20260619 14:45:15.765251 12746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:45:15.789234 12763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:15.790834 12746 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260619 14:45:15.791021 12746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "ac86cab385c444e198db095959aba51b"
format_stamp: "Formatted at 2026-06-19 14:45:15 on dist-test-slave-cr22"
server_key: "2a358950f56756cd4ac4d69a567c3286"
server_key_iv: "6475b26e46b6880bafc499ba526374c6"
server_key_version: "kuduclusterkey@0"
I20260619 14:45:15.791178 12746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:15.809712 12746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:15.813723 12746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:15.814036 12746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:15.814973 12746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:45:15.816224 12746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:45:15.816337 12746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:15.816496 12746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:45:15.816601 12746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:15.831223 12746 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43949
I20260619 14:45:15.832674 12746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:45:15.840794 12876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43949 every 8 connection(s)
I20260619 14:45:15.842216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 12746
I20260619 14:45:15.842381 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:45:15.842718 392 external_mini_cluster.cc:1428] Setting key 001fa37adf4d7ce760eefcb07c5618ac
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 19 14:45:15 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:45:15.870440 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:15.851382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45175 (local address 127.0.98.1:53873)
0619 14:45:15.853222 (+ 1840us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:15.853287 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:15.854480 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:15.855339 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:15.855349 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:15.855781 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:15.856547 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:15.856568 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:15.857910 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:15.857913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:15.859610 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:15.859622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:15.860634 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:15.860638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:15.860923 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:15.861721 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:15.861740 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:15.863529 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:15.866231 (+ 2702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:15.866239 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:15.866261 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:15.866690 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:15.867109 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:15.867113 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:15.867115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:15.867324 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:15.867939 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:15.867949 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:15.868344 (+ 395us) client_negotiation.cc:769] Sending connection context
0619 14:45:15.868589 (+ 245us) client_negotiation.cc:240] Negotiation successful
0619 14:45:15.869222 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1586,"thread_start_us":142,"threads_started":1}
I20260619 14:45:15.871595 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:15.850836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53873 (local address 127.0.98.62:45175)
0619 14:45:15.852897 (+ 2061us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:15.852906 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:15.853403 (+ 497us) server_negotiation.cc:407] Connection header received
0619 14:45:15.854754 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:15.854759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:15.854821 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:15.854886 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:15.856744 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:15.857727 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:15.859771 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:15.860249 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:15.860404 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:15.863786 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:15.863794 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:15.863801 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:15.863841 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:15.866068 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:15.866842 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:15.866846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:15.866850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:15.866967 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:15.867492 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:15.867496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:15.867499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:15.867774 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:15.867877 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:15.871200 (+ 3323us) server_negotiation.cc:299] Negotiation successful
0619 14:45:15.871362 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1962,"thread_start_us":152,"threads_started":1}
I20260619 14:45:15.873564 12877 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45175
I20260619 14:45:15.873935 12877 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:15.874568 12877 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
I20260619 14:45:15.876446 12664 ts_manager.cc:194] Registered new tserver with Master: ac86cab385c444e198db095959aba51b (127.0.98.1:43949)
I20260619 14:45:15.877993 12664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:45:15.883258 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43053/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45175
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:45:15 dist-test-slave-cr22 krb5kdc[12188](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 14:45:16.062749 12885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:16.063202 12885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:16.063308 12885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:16.070147 12885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:16.070289 12885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:16.070422 12885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:45:16.078388 12885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:43053/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:16.080121 12885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:16.081576 12885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:16.094514 12891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:16.094842 12894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:16.095288 12885 server_base.cc:1034] running on GCE node
W20260619 14:45:16.098232 12892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:16.098943 12885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:16.099861 12885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:16.101078 12885 hybrid_clock.cc:648] HybridClock initialized: now 1781880316101005 us; error 91 us; skew 500 ppm
Jun 19 14:45:16 dist-test-slave-cr22 krb5kdc[12188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:16.105177 12885 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:45:16.106549 12885 webserver.cc:460] Webserver started at http://127.0.98.2:40925/ using document root <none> and password file <none>
I20260619 14:45:16.107344 12885 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:16.107482 12885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:16.107811 12885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:45:16 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:45:16.126596 12885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "b05b1b52cb4d4f5597ce0992b574c3b4"
format_stamp: "Formatted at 2026-06-19 14:45:16 on dist-test-slave-cr22"
server_key: "4018ce8f9c86d0f42460e6567f7e0d15"
server_key_iv: "c77e72462897725c9846662adb20084e"
server_key_version: "kuduclusterkey@0"
I20260619 14:45:16.127411 12885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b05b1b52cb4d4f5597ce0992b574c3b4"
format_stamp: "Formatted at 2026-06-19 14:45:16 on dist-test-slave-cr22"
server_key: "4018ce8f9c86d0f42460e6567f7e0d15"
server_key_iv: "c77e72462897725c9846662adb20084e"
server_key_version: "kuduclusterkey@0"
I20260619 14:45:16.132265 12885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:45:16.155880 12901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:16.157747 12885 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260619 14:45:16.157959 12885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "b05b1b52cb4d4f5597ce0992b574c3b4"
format_stamp: "Formatted at 2026-06-19 14:45:16 on dist-test-slave-cr22"
server_key: "4018ce8f9c86d0f42460e6567f7e0d15"
server_key_iv: "c77e72462897725c9846662adb20084e"
server_key_version: "kuduclusterkey@0"
I20260619 14:45:16.158200 12885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:16.182493 12885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:16.186204 12885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:16.186460 12885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:16.187197 12885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:45:16.188333 12885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:45:16.188546 12885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:16.188668 12885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:45:16.188735 12885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:16.199146 12885 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34519
I20260619 14:45:16.201069 12885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:45:16.205968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 12885
I20260619 14:45:16.206108 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:45:16.206418 392 external_mini_cluster.cc:1428] Setting key 6a32e4a5b6acfade0e4acc7c5554273f
I20260619 14:45:16.206192 13014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34519 every 8 connection(s)
Jun 19 14:45:16 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:45:16.219238 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:16.203233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34687 (local address 127.0.98.62:45175)
0619 14:45:16.203408 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:16.203415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:16.204444 (+ 1029us) server_negotiation.cc:407] Connection header received
0619 14:45:16.205885 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:16.205901 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:16.205960 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:16.206018 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:16.207721 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:16.208658 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:16.210502 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:16.210913 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:16.211002 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:16.214298 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:16.214303 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:16.214306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:16.214348 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:16.216398 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:16.217157 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:16.217162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:16.217164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:16.217255 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:16.217685 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:16.217689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:16.217691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:16.217948 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:16.218035 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:16.218833 (+ 798us) server_negotiation.cc:299] Negotiation successful
0619 14:45:16.219016 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 14:45:16.221186 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:16.203691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45175 (local address 127.0.98.2:34687)
0619 14:45:16.204266 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:16.204328 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:16.205611 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:16.206447 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:16.206458 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:16.206828 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:16.207541 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:16.207560 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:16.208831 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:16.208835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:16.210349 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:16.210361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:16.211218 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:16.211222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:16.211457 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:16.212165 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:16.212187 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:16.214088 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:16.216589 (+ 2501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:16.216596 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:16.216616 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:16.217029 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:16.217378 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:16.217382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:16.217384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:16.217573 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:16.218107 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:16.218155 (+ 48us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:16.218523 (+ 368us) client_negotiation.cc:769] Sending connection context
0619 14:45:16.219431 (+ 908us) client_negotiation.cc:240] Negotiation successful
0619 14:45:16.219833 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":137,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:45:16.222638 13015 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45175
I20260619 14:45:16.222903 13015 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:16.223543 13015 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
I20260619 14:45:16.224855 12664 ts_manager.cc:194] Registered new tserver with Master: b05b1b52cb4d4f5597ce0992b574c3b4 (127.0.98.2:34519)
I20260619 14:45:16.225737 12664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:45:16.262909 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43053/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45175
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:45:16 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 14:45:16.425272 13020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:16.425652 13020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:16.425714 13020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:16.431851 13020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:16.431934 13020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:16.432019 13020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:45:16.439879 13020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:43053/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:16.441455 13020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:16.442788 13020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:16.454903 13026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:16.455047 13020 server_base.cc:1034] running on GCE node
W20260619 14:45:16.455271 13025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:16.455560 13028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:16.456008 13020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:16.456809 13020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:16.457990 13020 hybrid_clock.cc:648] HybridClock initialized: now 1781880316457968 us; error 37 us; skew 500 ppm
Jun 19 14:45:16 dist-test-slave-cr22 krb5kdc[12188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:16.465639 13020 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:45:16.466574 13020 webserver.cc:460] Webserver started at http://127.0.98.3:35611/ using document root <none> and password file <none>
I20260619 14:45:16.467229 13020 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:16.467326 13020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:16.467638 13020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:45:16 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:45:16.481328 13020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "b3fff158beff4f7795c18a4958321d64"
format_stamp: "Formatted at 2026-06-19 14:45:16 on dist-test-slave-cr22"
server_key: "807d64678c34b4c8de9a2f48b96d97f3"
server_key_iv: "eb97fcad11b0041e495f6252d531608e"
server_key_version: "kuduclusterkey@0"
I20260619 14:45:16.482213 13020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b3fff158beff4f7795c18a4958321d64"
format_stamp: "Formatted at 2026-06-19 14:45:16 on dist-test-slave-cr22"
server_key: "807d64678c34b4c8de9a2f48b96d97f3"
server_key_iv: "eb97fcad11b0041e495f6252d531608e"
server_key_version: "kuduclusterkey@0"
I20260619 14:45:16.487468 13020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 14:45:16.517277 13035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:16.519160 13020 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:45:16.519345 13020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "b3fff158beff4f7795c18a4958321d64"
format_stamp: "Formatted at 2026-06-19 14:45:16 on dist-test-slave-cr22"
server_key: "807d64678c34b4c8de9a2f48b96d97f3"
server_key_iv: "eb97fcad11b0041e495f6252d531608e"
server_key_version: "kuduclusterkey@0"
I20260619 14:45:16.519595 13020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:16.569490 13020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:16.573468 13020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:16.573706 13020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:16.574502 13020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:45:16.575637 13020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:45:16.575708 13020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:16.575834 13020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:45:16.575878 13020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:16.587919 13020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39379
I20260619 14:45:16.589180 13020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:45:16.590107 13148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39379 every 8 connection(s)
I20260619 14:45:16.596029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 13020
I20260619 14:45:16.596161 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:45:16.596478 392 external_mini_cluster.cc:1428] Setting key aa574e4da61e9ee2f4b005629347bdd9
Jun 19 14:45:16 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:45:16.608099 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:16.591225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34647 (local address 127.0.98.62:45175)
0619 14:45:16.591368 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:16.591373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:16.592620 (+ 1247us) server_negotiation.cc:407] Connection header received
0619 14:45:16.594020 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:16.594025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:16.594121 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:16.594223 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:16.597695 (+ 3472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:16.598521 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:16.600368 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:16.600707 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:16.600804 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:16.604010 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:16.604014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:16.604017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:16.604045 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:16.605701 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:16.606314 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:16.606318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:16.606320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:16.606401 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:16.606813 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:16.606816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:16.606818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:16.607037 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:16.607133 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:16.607819 (+ 686us) server_negotiation.cc:299] Negotiation successful
0619 14:45:16.607935 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:45:16.609143 13151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:16.591854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45175 (local address 127.0.98.3:34647)
0619 14:45:16.592455 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:16.592511 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:16.593755 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:16.594428 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:16.594438 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:16.594805 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:16.595655 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:16.595676 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:16.598646 (+ 2970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:16.598651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:16.600228 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:16.600241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:16.600972 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:16.600976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:16.601188 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:16.601869 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:16.601890 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:16.603804 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:16.605826 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:16.605835 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:16.605850 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:16.606198 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:16.606504 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:16.606507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:16.606509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:16.606702 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:16.607159 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:16.607166 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:16.607541 (+ 375us) client_negotiation.cc:769] Sending connection context
0619 14:45:16.607748 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 14:45:16.608096 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":161,"threads_started":1}
I20260619 14:45:16.610525 13149 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45175
I20260619 14:45:16.611083 13149 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:16.611783 13149 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
I20260619 14:45:16.612864 12664 ts_manager.cc:194] Registered new tserver with Master: b3fff158beff4f7795c18a4958321d64 (127.0.98.3:39379)
I20260619 14:45:16.613592 12664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34647
I20260619 14:45:16.623638 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:45:16.624536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 12746
I20260619 14:45:16.631130 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 12885
I20260619 14:45:16.638886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 13020
I20260619 14:45:16.647087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 12632
I20260619 14:45:16.652444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45175
--webserver_interface=127.0.98.62
--webserver_port=36361
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45175
--ranger_config_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43053/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43053/kms with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:45:16.768507 13152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:16.768796 13152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:16.768837 13152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:16.773135 13152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:45:16.773213 13152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:16.773234 13152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:16.773250 13152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:45:16.773267 13152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:45:16.779198 13152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43053/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45175
--ranger_config_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:16.780354 13152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:16.781397 13152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:16.790241 13157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:16.790244 13160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:16.790839 13158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:16.791054 13152 server_base.cc:1034] running on GCE node
I20260619 14:45:16.791705 13152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:16.792922 13152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:16.794133 13152 hybrid_clock.cc:648] HybridClock initialized: now 1781880316794098 us; error 47 us; skew 500 ppm
Jun 19 14:45:16 dist-test-slave-cr22 krb5kdc[12188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:16.797372 13152 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:45:16.798413 13152 webserver.cc:460] Webserver started at http://127.0.98.62:36361/ using document root <none> and password file <none>
I20260619 14:45:16.798959 13152 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:16.799026 13152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:45:16 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:45:16.854945 13152 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:45:16.857422 13168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:16.858768 13152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:45:16.858913 13152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "08ab7d33e6804e54b64770e789b36ca6"
format_stamp: "Formatted at 2026-06-19 14:45:14 on dist-test-slave-cr22"
server_key: "a2d3e8a8f0bce433634ddd8cfe940d41"
server_key_iv: "a3a9b5d229ac0946bc1e61c4a8a41cad"
server_key_version: "kuduclusterkey@0"
I20260619 14:45:16.859650 13152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:16.871651 13152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:16.874418 13152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:16.874701 13152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:16.882295 13152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45175
I20260619 14:45:16.882311 13220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45175 every 8 connection(s)
I20260619 14:45:16.883348 13152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:45:16.887779 13221 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:45:16.889554 13221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6: Bootstrap starting.
I20260619 14:45:16.889891 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 13152
I20260619 14:45:16.890520 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43053/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43949
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45175
--webserver_port=38135
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43053/kms with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:45:16.906688 13221 log.cc:824] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:16.913967 13221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:45:16.914386 13221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6: Bootstrap complete.
I20260619 14:45:16.918748 13221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ab7d33e6804e54b64770e789b36ca6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } }
I20260619 14:45:16.919137 13221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ab7d33e6804e54b64770e789b36ca6, State: Initialized, Role: FOLLOWER
I20260619 14:45:16.919687 13221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ab7d33e6804e54b64770e789b36ca6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } }
I20260619 14:45:16.919806 13221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:45:16.919893 13221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:45:16.919989 13221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:45:16.921885 13221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ab7d33e6804e54b64770e789b36ca6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } }
I20260619 14:45:16.922222 13221 leader_election.cc:302] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08ab7d33e6804e54b64770e789b36ca6; no voters:
I20260619 14:45:16.922557 13221 leader_election.cc:288] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:45:16.922642 13225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:45:16.922837 13225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [term 2 LEADER]: Becoming Leader. State: Replica: 08ab7d33e6804e54b64770e789b36ca6, State: Running, Role: LEADER
I20260619 14:45:16.923130 13225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ab7d33e6804e54b64770e789b36ca6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } }
I20260619 14:45:16.923823 13221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:45:16.924479 13226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "08ab7d33e6804e54b64770e789b36ca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ab7d33e6804e54b64770e789b36ca6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } } }
I20260619 14:45:16.924616 13226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:45:16.925326 13227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08ab7d33e6804e54b64770e789b36ca6. Latest consensus state: current_term: 2 leader_uuid: "08ab7d33e6804e54b64770e789b36ca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ab7d33e6804e54b64770e789b36ca6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45175 } } }
I20260619 14:45:16.925472 13227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:45:16.925771 13232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:45:16.926173 13221 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:45:16.932121 13232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:45:16.933035 13232 catalog_manager.cc:1259] Loaded cluster ID: f44b40a8eaf944a8bce34b9d33b2a79b
I20260619 14:45:16.933101 13232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:45:16.936179 13232 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:45:16.936945 13232 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6: Loaded TSK: 0
I20260619 14:45:16.938279 13232 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:45:17.035501 13223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:17.035876 13223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:17.035953 13223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:17.042088 13223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:17.042196 13223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:17.042286 13223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:45:17.049059 13223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43053/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38135
--tserver_master_addrs=127.0.98.62:45175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:17.050489 13223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:17.051810 13223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:17.062937 13259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:17.062940 13261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:17.063052 13258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:17.063916 13223 server_base.cc:1034] running on GCE node
I20260619 14:45:17.064337 13223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:17.065060 13223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:17.066473 13223 hybrid_clock.cc:648] HybridClock initialized: now 1781880317066359 us; error 51 us; skew 500 ppm
Jun 19 14:45:17 dist-test-slave-cr22 krb5kdc[12188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:17.070153 13223 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:45:17.071290 13223 webserver.cc:460] Webserver started at http://127.0.98.1:38135/ using document root <none> and password file <none>
I20260619 14:45:17.072062 13223 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:17.072186 13223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:45:17 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:45:17.096366 13223 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260619 14:45:17.100176 13269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:17.102293 13223 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 14:45:17.102447 13223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "ac86cab385c444e198db095959aba51b"
format_stamp: "Formatted at 2026-06-19 14:45:15 on dist-test-slave-cr22"
server_key: "2a358950f56756cd4ac4d69a567c3286"
server_key_iv: "6475b26e46b6880bafc499ba526374c6"
server_key_version: "kuduclusterkey@0"
I20260619 14:45:17.103266 13223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:17.115856 13223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:17.119252 13223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:17.119546 13223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:17.120255 13223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:45:17.121135 13223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:45:17.121212 13223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:17.121285 13223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:45:17.121344 13223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:17.133882 13223 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43949
I20260619 14:45:17.135265 13223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:45:17.135752 13382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43949 every 8 connection(s)
I20260619 14:45:17.140389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 13223
I20260619 14:45:17.140807 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43053/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34519
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45175
--webserver_port=40925
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43053/kms with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:45:17 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:45:17.160856 13386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:17.139925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45175 (local address 127.0.98.1:44537)
0619 14:45:17.141095 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:17.141150 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:17.142152 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:17.144202 (+ 2050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:17.144214 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:17.144609 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:17.145352 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:17.145377 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:17.147727 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:17.147733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:17.149604 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:17.149619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:17.151178 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:17.151185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:17.151477 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:17.152279 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:17.152309 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:17.154245 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:17.157183 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:17.157191 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:17.157205 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:17.157550 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:17.157963 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:17.157967 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:17.157969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:17.158133 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:17.158900 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:17.158907 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:17.159204 (+ 297us) client_negotiation.cc:769] Sending connection context
0619 14:45:17.159396 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 14:45:17.159786 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"thread_start_us":146,"threads_started":1}
I20260619 14:45:17.162364 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:17.139681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44537 (local address 127.0.98.62:45175)
0619 14:45:17.140254 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:17.140280 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:17.141299 (+ 1019us) server_negotiation.cc:407] Connection header received
0619 14:45:17.142812 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:17.142830 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:17.143284 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:17.143687 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:17.145598 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:17.147530 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:17.149778 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:17.150703 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:17.150904 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:17.154492 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:17.154500 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:17.154517 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:17.154809 (+ 292us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:17.157036 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:17.157707 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:17.157714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:17.157719 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:17.157860 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:17.158232 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:17.158235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:17.158237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:17.158723 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:17.158838 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:17.161216 (+ 2378us) server_negotiation.cc:299] Negotiation successful
0619 14:45:17.161515 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":131,"threads_started":1}
I20260619 14:45:17.165736 13383 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45175
I20260619 14:45:17.166009 13383 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:17.166625 13383 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
W20260619 14:45:17.168903 13383 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45175 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 14:45:17.169039 13383 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:17.169230 13383 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
I20260619 14:45:17.170101 13383 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:17.170279 13383 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
W20260619 14:45:17.171010 13383 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:45:17.288630 13387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:17.289019 13387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:17.289106 13387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:17.293956 13387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:17.294047 13387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:17.294155 13387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:45:17.300050 13387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43053/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40925
--tserver_master_addrs=127.0.98.62:45175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:17.301236 13387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:17.302408 13387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:17.312966 13392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:17.313298 13395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:17.313294 13387 server_base.cc:1034] running on GCE node
W20260619 14:45:17.312932 13393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:17.314158 13387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:17.314988 13387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:17.316184 13387 hybrid_clock.cc:648] HybridClock initialized: now 1781880317316159 us; error 36 us; skew 500 ppm
Jun 19 14:45:17 dist-test-slave-cr22 krb5kdc[12188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:17.319355 13387 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:45:17.320266 13387 webserver.cc:460] Webserver started at http://127.0.98.2:40925/ using document root <none> and password file <none>
I20260619 14:45:17.320952 13387 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:17.321056 13387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:45:17 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:45:17.342187 13387 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260619 14:45:17.345654 13403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:17.347296 13387 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 14:45:17.347431 13387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "b05b1b52cb4d4f5597ce0992b574c3b4"
format_stamp: "Formatted at 2026-06-19 14:45:16 on dist-test-slave-cr22"
server_key: "4018ce8f9c86d0f42460e6567f7e0d15"
server_key_iv: "c77e72462897725c9846662adb20084e"
server_key_version: "kuduclusterkey@0"
I20260619 14:45:17.348171 13387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:17.364557 13387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:17.367949 13387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:17.368151 13387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:17.368902 13387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:45:17.369884 13387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:45:17.369936 13387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:17.370002 13387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:45:17.370045 13387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:17.381003 13387 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34519
I20260619 14:45:17.382200 13387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:45:17.383966 13516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34519 every 8 connection(s)
I20260619 14:45:17.392207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 13387
I20260619 14:45:17.392767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43053/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39379
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45175
--webserver_port=35611
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43053/kms with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:45:17 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:45:17.399755 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:17.385878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38093 (local address 127.0.98.62:45175)
0619 14:45:17.386019 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:17.386023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:17.387225 (+ 1202us) server_negotiation.cc:407] Connection header received
0619 14:45:17.388488 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:17.388492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:17.388570 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:17.388682 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:17.390108 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:17.391010 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:17.392496 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:17.392732 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:17.392857 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:17.395534 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:17.395538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:17.395542 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:17.395576 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:17.397398 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:17.397950 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:17.397954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:17.397956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:17.398046 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:17.398445 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:17.398450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:17.398452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:17.398651 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:17.398765 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:17.399414 (+ 649us) server_negotiation.cc:299] Negotiation successful
0619 14:45:17.399581 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:45:17.400715 13519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:17.386453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45175 (local address 127.0.98.2:38093)
0619 14:45:17.387057 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:17.387116 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:17.388230 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:17.388869 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:17.388878 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:17.389228 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:17.389910 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:17.389932 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:17.391173 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:17.391176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:17.392274 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:17.392284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:17.392900 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:17.392904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:17.393135 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:17.393795 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:17.393812 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:17.395285 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:17.397546 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:17.397552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:17.397577 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:17.397839 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:17.398143 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:17.398148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:17.398150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:17.398350 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:17.398821 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:17.398827 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:17.399147 (+ 320us) client_negotiation.cc:769] Sending connection context
0619 14:45:17.399375 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 14:45:17.399718 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":157,"threads_started":1}
I20260619 14:45:17.401880 13517 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45175
I20260619 14:45:17.402065 13517 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:17.402678 13517 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
W20260619 14:45:17.404472 13517 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45175 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 14:45:17.404609 13517 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:17.404772 13517 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
I20260619 14:45:17.405560 13517 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:17.405747 13517 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
W20260619 14:45:17.406478 13517 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:45:17.557789 13520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:17.558066 13520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:17.558112 13520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:17.562562 13520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:17.562655 13520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:17.562762 13520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:45:17.569024 13520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43053/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35611
--tserver_master_addrs=127.0.98.62:45175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:17.570463 13520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:17.571789 13520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:17.584762 13527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:17.584863 13526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:17.585386 13529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:17.585737 13520 server_base.cc:1034] running on GCE node
I20260619 14:45:17.586395 13520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:17.587260 13520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:17.588464 13520 hybrid_clock.cc:648] HybridClock initialized: now 1781880317588437 us; error 43 us; skew 500 ppm
Jun 19 14:45:17 dist-test-slave-cr22 krb5kdc[12188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:17.592159 13520 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:45:17.593268 13520 webserver.cc:460] Webserver started at http://127.0.98.3:35611/ using document root <none> and password file <none>
I20260619 14:45:17.593961 13520 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:17.594060 13520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:45:17 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:45:17.617089 13520 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260619 14:45:17.620486 13537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:17.622278 13520 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:45:17.622443 13520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "b3fff158beff4f7795c18a4958321d64"
format_stamp: "Formatted at 2026-06-19 14:45:16 on dist-test-slave-cr22"
server_key: "807d64678c34b4c8de9a2f48b96d97f3"
server_key_iv: "eb97fcad11b0041e495f6252d531608e"
server_key_version: "kuduclusterkey@0"
I20260619 14:45:17.623291 13520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:17.645203 13520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:17.648878 13520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:17.649101 13520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:17.649708 13520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:45:17.650547 13520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:45:17.650616 13520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:17.650684 13520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:45:17.650733 13520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:17.661427 13520 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39379
I20260619 14:45:17.661460 13650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39379 every 8 connection(s)
I20260619 14:45:17.662765 13520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:45:17.672807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 13520
Jun 19 14:45:17 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:45:17.681821 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:17.666510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45175 (local address 127.0.98.3:33267)
0619 14:45:17.667030 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:17.667086 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:17.668092 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:17.668702 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:17.668710 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:17.669106 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:17.669828 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:17.669850 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:17.671037 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:17.671041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:17.672380 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:17.672390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:17.673205 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:17.673210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:17.673408 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:17.674190 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:17.674212 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:17.676058 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:17.678358 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:17.678365 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:17.678381 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:17.678740 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:17.679082 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:17.679085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:17.679087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:17.679270 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:17.679833 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:17.679839 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:17.680187 (+ 348us) client_negotiation.cc:769] Sending connection context
0619 14:45:17.680400 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 14:45:17.680765 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":155,"threads_started":1}
I20260619 14:45:17.682855 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:17.665980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33267 (local address 127.0.98.62:45175)
0619 14:45:17.666125 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:17.666130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:17.667195 (+ 1065us) server_negotiation.cc:407] Connection header received
0619 14:45:17.668322 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:17.668325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:17.668384 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:17.668509 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:17.670020 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:17.670870 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:17.672520 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:17.672891 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:17.673039 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:17.676253 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:17.676256 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:17.676259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:17.676282 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:17.678216 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:17.678862 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:17.678865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:17.678868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:17.678959 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:17.679403 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:17.679407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:17.679409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:17.679672 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:17.679833 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:17.682565 (+ 2732us) server_negotiation.cc:299] Negotiation successful
0619 14:45:17.682707 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:45:17.683836 13651 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45175
I20260619 14:45:17.684087 13651 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:17.684649 13651 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
I20260619 14:45:17.687060 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:17.673667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.0.98.62:45175)
0619 14:45:17.674323 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:17.674326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:17.674337 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:45:17.674368 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:17.674370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:17.674408 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:17.674527 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:17.675489 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:17.676819 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:17.678543 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:17.678932 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:17.679056 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:17.681564 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:17.681569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:17.681572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:17.681598 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:17.684364 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:17.684937 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:17.684940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:17.684942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:17.685046 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:17.685388 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:17.685391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:17.685393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:17.685609 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:17.686551 (+ 942us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:17.686642 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 14:45:17.686784 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":442,"threads_started":1}
W20260619 14:45:17.689934 13651 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45175 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 14:45:17.690075 13651 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:17.690259 13651 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
I20260619 14:45:17.691179 13651 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:17.691387 13651 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
W20260619 14:45:17.692709 13651 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 14:45:18 dist-test-slave-cr22 krb5kdc[12188](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:18.171231 13383 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:18.171594 13383 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
I20260619 14:45:18.172837 13184 ts_manager.cc:194] Registered new tserver with Master: ac86cab385c444e198db095959aba51b (127.0.98.1:43949)
I20260619 14:45:18.174494 13184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44537
I20260619 14:45:18.406733 13517 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:18.407114 13517 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
I20260619 14:45:18.408149 13184 ts_manager.cc:194] Registered new tserver with Master: b05b1b52cb4d4f5597ce0992b574c3b4 (127.0.98.2:34519)
I20260619 14:45:18.408874 13184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38093
Jun 19 14:45:18 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:45:18.692977 13651 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:18.693280 13651 heartbeater.cc:511] Master 127.0.98.62:45175 requested a full tablet report, sending...
I20260619 14:45:18.694043 13185 ts_manager.cc:194] Registered new tserver with Master: b3fff158beff4f7795c18a4958321d64 (127.0.98.3:39379)
I20260619 14:45:18.694830 13185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33267
I20260619 14:45:18.702224 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:45:18.714105 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:18.703918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53138 (local address 127.0.98.62:45175)
0619 14:45:18.704627 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:18.704631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:18.704644 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:45:18.704681 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:18.704683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:18.704727 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:18.704806 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:18.705760 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:18.706582 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:18.707762 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:18.708109 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:18.708203 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:18.709942 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:18.709946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:18.709949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:18.709977 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:18.711917 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:18.712391 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:18.712395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:18.712397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:18.712481 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:18.712752 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:18.712755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:18.712756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:18.712955 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:18.713085 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:18.713799 (+ 714us) server_negotiation.cc:299] Negotiation successful
0619 14:45:18.713923 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":81,"threads_started":1}
I20260619 14:45:18.718472 13185 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:45:18.723457 13185 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:45:18.748451 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:18.737332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39379 (local address 127.0.0.1:48680)
0619 14:45:18.738773 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:18.738824 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:18.738981 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:18.740492 (+ 1511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:18.740495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:18.740513 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:18.740816 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:18.740821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:18.744872 (+ 4051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:18.744875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:18.746306 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:18.746317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:18.747806 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:18.747810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:18.747929 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:18.747980 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 14:45:18.748117 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 14:45:18.748274 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1260,"mutex_wait_us":186,"thread_start_us":105,"threads_started":1}
I20260619 14:45:18.750708 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:18.738093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34519 (local address 127.0.0.1:43976)
0619 14:45:18.740985 (+ 2892us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:18.741008 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:18.741085 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:18.741715 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:18.741718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:18.741734 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:18.741816 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:18.741822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:18.745495 (+ 3673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:18.745499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:18.746727 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:18.746739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:18.750313 (+ 3574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:18.750316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:18.750405 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:18.750427 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:45:18.750488 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 14:45:18.750562 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2836,"thread_start_us":77,"threads_started":1}
I20260619 14:45:18.751811 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:18.737448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48680 (local address 127.0.98.3:39379)
0619 14:45:18.739616 (+ 2168us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:18.739629 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:18.739890 (+ 261us) server_negotiation.cc:407] Connection header received
0619 14:45:18.739973 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:18.739978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:18.740168 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:18.740327 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:18.743575 (+ 3248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:18.744735 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:18.746465 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:18.747421 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:18.747577 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:18.747742 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:18.751542 (+ 3800us) server_negotiation.cc:299] Negotiation successful
0619 14:45:18.751669 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2058,"thread_start_us":109,"threads_started":1}
I20260619 14:45:18.753108 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:18.738491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.0.98.2:34519)
0619 14:45:18.739113 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:18.739120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:18.741179 (+ 2059us) server_negotiation.cc:407] Connection header received
0619 14:45:18.741257 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:18.741261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:18.741436 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:18.741568 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:18.741944 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:18.745373 (+ 3429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:18.747794 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:18.748648 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:18.748770 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:18.748909 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:18.752838 (+ 3929us) server_negotiation.cc:299] Negotiation successful
0619 14:45:18.752954 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":96,"threads_started":1}
I20260619 14:45:18.754369 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:18.737740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43949 (local address 127.0.0.1:41574)
0619 14:45:18.739247 (+ 1507us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:18.739259 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:18.739355 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:18.740457 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:18.740463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:18.740488 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:18.740676 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:18.740683 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:18.749035 (+ 8352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:18.749039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:18.750171 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:18.750183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:18.753949 (+ 3766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:18.753953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:18.754067 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:18.754093 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:45:18.754162 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 14:45:18.754229 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1446,"mutex_wait_us":1,"spinlock_wait_cycles":6016,"thread_start_us":89,"threads_started":1}
I20260619 14:45:18.755270 13581 tablet_service.cc:1467] Processing CreateTablet for tablet 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 (DEFAULT_TABLE table=test-table [id=a383d5b5ad7a41edac6571c8d2977b5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:45:18.756515 13581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cfb57c5ed0543ed8bb3bb8cb71c4bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:18.757635 13446 tablet_service.cc:1467] Processing CreateTablet for tablet 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 (DEFAULT_TABLE table=test-table [id=a383d5b5ad7a41edac6571c8d2977b5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:45:18.758677 13446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cfb57c5ed0543ed8bb3bb8cb71c4bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:18.767969 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:18.738169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.0.98.1:43949)
0619 14:45:18.739356 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:18.739362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:18.739388 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:45:18.739488 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:18.739494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:18.739716 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:18.743689 (+ 3973us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:18.743860 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:18.744981 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:18.751020 (+ 6039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:18.753607 (+ 2587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:18.753716 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:18.767593 (+ 13877us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:18.767666 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 14:45:18.767795 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1088,"thread_start_us":112,"threads_started":1}
I20260619 14:45:18.771337 13685 tablet_bootstrap.cc:492] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4: Bootstrap starting.
I20260619 14:45:18.771129 13309 tablet_service.cc:1467] Processing CreateTablet for tablet 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 (DEFAULT_TABLE table=test-table [id=a383d5b5ad7a41edac6571c8d2977b5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:45:18.771788 13684 tablet_bootstrap.cc:492] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64: Bootstrap starting.
I20260619 14:45:18.772315 13309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cfb57c5ed0543ed8bb3bb8cb71c4bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:18.773721 13685 tablet_bootstrap.cc:654] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4: Neither blocks nor log segments found. Creating new log.
I20260619 14:45:18.773823 13684 tablet_bootstrap.cc:654] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64: Neither blocks nor log segments found. Creating new log.
I20260619 14:45:18.774573 13685 log.cc:824] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:18.774585 13684 log.cc:824] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:18.776988 13684 tablet_bootstrap.cc:492] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64: No bootstrap required, opened a new log
I20260619 14:45:18.777204 13684 ts_tablet_manager.cc:1397] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20260619 14:45:18.777676 13685 tablet_bootstrap.cc:492] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4: No bootstrap required, opened a new log
I20260619 14:45:18.777846 13685 ts_tablet_manager.cc:1397] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 14:45:18.781966 13684 raft_consensus.cc:348] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fff158beff4f7795c18a4958321d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } } peers { permanent_uuid: "b05b1b52cb4d4f5597ce0992b574c3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34519 } } peers { permanent_uuid: "ac86cab385c444e198db095959aba51b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43949 } }
I20260619 14:45:18.781987 13685 raft_consensus.cc:348] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fff158beff4f7795c18a4958321d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } } peers { permanent_uuid: "b05b1b52cb4d4f5597ce0992b574c3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34519 } } peers { permanent_uuid: "ac86cab385c444e198db095959aba51b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43949 } }
I20260619 14:45:18.782259 13685 raft_consensus.cc:374] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:45:18.782315 13685 raft_consensus.cc:729] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b05b1b52cb4d4f5597ce0992b574c3b4, State: Initialized, Role: FOLLOWER
I20260619 14:45:18.782536 13684 raft_consensus.cc:374] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:45:18.782719 13684 raft_consensus.cc:729] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3fff158beff4f7795c18a4958321d64, State: Initialized, Role: FOLLOWER
I20260619 14:45:18.782830 13685 consensus_queue.cc:260] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fff158beff4f7795c18a4958321d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } } peers { permanent_uuid: "b05b1b52cb4d4f5597ce0992b574c3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34519 } } peers { permanent_uuid: "ac86cab385c444e198db095959aba51b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43949 } }
I20260619 14:45:18.783342 13684 consensus_queue.cc:260] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fff158beff4f7795c18a4958321d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } } peers { permanent_uuid: "b05b1b52cb4d4f5597ce0992b574c3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34519 } } peers { permanent_uuid: "ac86cab385c444e198db095959aba51b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43949 } }
I20260619 14:45:18.783952 13685 ts_tablet_manager.cc:1428] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:45:18.784328 13517 heartbeater.cc:503] Master 127.0.98.62:45175 was elected leader, sending a full tablet report...
I20260619 14:45:18.784559 13684 ts_tablet_manager.cc:1428] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260619 14:45:18.784929 13651 heartbeater.cc:503] Master 127.0.98.62:45175 was elected leader, sending a full tablet report...
I20260619 14:45:18.790601 13690 tablet_bootstrap.cc:492] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P ac86cab385c444e198db095959aba51b: Bootstrap starting.
I20260619 14:45:18.792845 13690 tablet_bootstrap.cc:654] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P ac86cab385c444e198db095959aba51b: Neither blocks nor log segments found. Creating new log.
I20260619 14:45:18.793655 13690 log.cc:824] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P ac86cab385c444e198db095959aba51b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:18.795703 13690 tablet_bootstrap.cc:492] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P ac86cab385c444e198db095959aba51b: No bootstrap required, opened a new log
I20260619 14:45:18.795926 13690 ts_tablet_manager.cc:1397] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P ac86cab385c444e198db095959aba51b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260619 14:45:18.799603 13690 raft_consensus.cc:348] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P ac86cab385c444e198db095959aba51b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fff158beff4f7795c18a4958321d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } } peers { permanent_uuid: "b05b1b52cb4d4f5597ce0992b574c3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34519 } } peers { permanent_uuid: "ac86cab385c444e198db095959aba51b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43949 } }
I20260619 14:45:18.799890 13690 raft_consensus.cc:374] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P ac86cab385c444e198db095959aba51b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:45:18.799984 13690 raft_consensus.cc:729] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P ac86cab385c444e198db095959aba51b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac86cab385c444e198db095959aba51b, State: Initialized, Role: FOLLOWER
I20260619 14:45:18.800486 13690 consensus_queue.cc:260] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P ac86cab385c444e198db095959aba51b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fff158beff4f7795c18a4958321d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } } peers { permanent_uuid: "b05b1b52cb4d4f5597ce0992b574c3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34519 } } peers { permanent_uuid: "ac86cab385c444e198db095959aba51b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43949 } }
I20260619 14:45:18.801189 13383 heartbeater.cc:503] Master 127.0.98.62:45175 was elected leader, sending a full tablet report...
I20260619 14:45:18.801610 13690 ts_tablet_manager.cc:1428] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P ac86cab385c444e198db095959aba51b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
Jun 19 14:45:18 dist-test-slave-cr22 krb5kdc[12188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 14:45:18.885783 13518 tablet.cc:2369] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:45:18.889210 13384 tablet.cc:2369] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P ac86cab385c444e198db095959aba51b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:45:18.915877 13652 tablet.cc:2369] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:45:19.001307 13689 raft_consensus.cc:482] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:45:19.001499 13688 raft_consensus.cc:482] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:45:19.001573 13689 raft_consensus.cc:504] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fff158beff4f7795c18a4958321d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } } peers { permanent_uuid: "b05b1b52cb4d4f5597ce0992b574c3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34519 } } peers { permanent_uuid: "ac86cab385c444e198db095959aba51b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43949 } }
I20260619 14:45:19.001742 13688 raft_consensus.cc:504] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fff158beff4f7795c18a4958321d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } } peers { permanent_uuid: "b05b1b52cb4d4f5597ce0992b574c3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34519 } } peers { permanent_uuid: "ac86cab385c444e198db095959aba51b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43949 } }
I20260619 14:45:19.003057 13689 leader_election.cc:288] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b05b1b52cb4d4f5597ce0992b574c3b4 (127.0.98.2:34519), ac86cab385c444e198db095959aba51b (127.0.98.1:43949)
I20260619 14:45:19.003067 13688 leader_election.cc:288] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3fff158beff4f7795c18a4958321d64 (127.0.98.3:39379), ac86cab385c444e198db095959aba51b (127.0.98.1:43949)
I20260619 14:45:19.014431 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:19.003777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41341 (local address 127.0.98.3:39379)
0619 14:45:19.004627 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:19.004633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:19.005634 (+ 1001us) server_negotiation.cc:407] Connection header received
0619 14:45:19.006098 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:19.006101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:19.006154 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:19.006203 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:19.007327 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:19.008242 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:19.012303 (+ 4061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:19.012904 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:19.013003 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:19.013078 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:19.014195 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0619 14:45:19.014274 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759}
I20260619 14:45:19.016273 13697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:19.003716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39379 (local address 127.0.98.2:41341)
0619 14:45:19.005537 (+ 1821us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:19.005554 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:19.005718 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:19.006349 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:19.006352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:19.006370 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:19.006541 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:19.006548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:19.008454 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:19.008457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:19.012190 (+ 3733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:19.012201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:19.013139 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:19.013142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:19.013256 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:19.013279 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:45:19.016006 (+ 2727us) client_negotiation.cc:240] Negotiation successful
0619 14:45:19.016099 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1750,"thread_start_us":118,"threads_started":1}
I20260619 14:45:19.016708 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:19.004143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43949 (local address 127.0.98.2:45201)
0619 14:45:19.012423 (+ 8280us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:19.012437 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:19.012521 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:19.012879 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:19.012882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:19.012897 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:19.013034 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:19.013041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:19.014068 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:19.014071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:19.015231 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:19.015241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:19.016255 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:19.016258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:19.016400 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:19.016427 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:45:19.016500 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 14:45:19.016571 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8213,"thread_start_us":97,"threads_started":1}
I20260619 14:45:19.016973 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfb57c5ed0543ed8bb3bb8cb71c4bf4" candidate_uuid: "b05b1b52cb4d4f5597ce0992b574c3b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3fff158beff4f7795c18a4958321d64" is_pre_election: true
I20260619 14:45:19.017088 13698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:19.004944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45201 (local address 127.0.98.1:43949)
0619 14:45:19.009091 (+ 4147us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:19.009101 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:19.012596 (+ 3495us) server_negotiation.cc:407] Connection header received
0619 14:45:19.012639 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:19.012642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:19.012685 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:19.012755 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:19.013150 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:19.013948 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:19.015363 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:19.016048 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:19.016134 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:19.016200 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:19.016863 (+ 663us) server_negotiation.cc:299] Negotiation successful
0619 14:45:19.016950 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4067,"thread_start_us":126,"threads_started":1}
I20260619 14:45:19.017232 13587 raft_consensus.cc:2454] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b05b1b52cb4d4f5597ce0992b574c3b4 in term 0.
I20260619 14:45:19.017740 13407 leader_election.cc:302] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b05b1b52cb4d4f5597ce0992b574c3b4, b3fff158beff4f7795c18a4958321d64; no voters:
I20260619 14:45:19.017813 13330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfb57c5ed0543ed8bb3bb8cb71c4bf4" candidate_uuid: "b05b1b52cb4d4f5597ce0992b574c3b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86cab385c444e198db095959aba51b" is_pre_election: true
I20260619 14:45:19.018119 13330 raft_consensus.cc:2454] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P ac86cab385c444e198db095959aba51b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b05b1b52cb4d4f5597ce0992b574c3b4 in term 0.
I20260619 14:45:19.018855 13688 raft_consensus.cc:2781] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:45:19.018972 13688 raft_consensus.cc:482] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:45:19.019037 13688 raft_consensus.cc:3037] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:45:19.019170 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:19.004711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48555 (local address 127.0.98.2:34519)
0619 14:45:19.004844 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:19.004849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:19.009382 (+ 4533us) server_negotiation.cc:407] Connection header received
0619 14:45:19.011133 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:19.011137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:19.011181 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:19.011245 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:19.013391 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:19.014077 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:19.017892 (+ 3815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:19.018468 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:19.018541 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:19.018595 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:19.018987 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 14:45:19.019071 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:45:19.019559 13699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:19.005181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34519 (local address 127.0.98.3:48555)
0619 14:45:19.009279 (+ 4098us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:19.009295 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:19.011021 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:19.011496 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:19.011498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:19.011515 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:19.011636 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:19.011642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:19.014715 (+ 3073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:19.014718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:19.015900 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:19.015910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:19.018691 (+ 2781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:19.018697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:19.018803 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:19.018828 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:45:19.019283 (+ 455us) client_negotiation.cc:240] Negotiation successful
0619 14:45:19.019365 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4026,"thread_start_us":92,"threads_started":1}
I20260619 14:45:19.020174 13470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfb57c5ed0543ed8bb3bb8cb71c4bf4" candidate_uuid: "b3fff158beff4f7795c18a4958321d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b05b1b52cb4d4f5597ce0992b574c3b4" is_pre_election: true
I20260619 14:45:19.020263 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:19.004846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46213 (local address 127.0.98.1:43949)
0619 14:45:19.004980 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:19.004984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:19.006001 (+ 1017us) server_negotiation.cc:407] Connection header received
0619 14:45:19.006802 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:19.006806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:19.006857 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:19.006907 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:19.009544 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:19.010441 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:19.018064 (+ 7623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:19.018640 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:19.018731 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:19.018783 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:19.020082 (+ 1299us) server_negotiation.cc:299] Negotiation successful
0619 14:45:19.020155 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:45:19.020599 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:19.003659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43949 (local address 127.0.98.3:46213)
0619 14:45:19.005916 (+ 2257us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:19.005930 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:19.006673 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:19.007037 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:19.007040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:19.007056 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:19.007191 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:19.007197 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:19.010596 (+ 3399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:19.010599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:19.017747 (+ 7148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:19.017760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:19.019823 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:19.019827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:19.019937 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:19.019961 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:45:19.020395 (+ 434us) client_negotiation.cc:240] Negotiation successful
0619 14:45:19.020472 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2191,"thread_start_us":155,"threads_started":1}
I20260619 14:45:19.020874 13688 raft_consensus.cc:504] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fff158beff4f7795c18a4958321d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } } peers { permanent_uuid: "b05b1b52cb4d4f5597ce0992b574c3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34519 } } peers { permanent_uuid: "ac86cab385c444e198db095959aba51b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43949 } }
I20260619 14:45:19.021121 13330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfb57c5ed0543ed8bb3bb8cb71c4bf4" candidate_uuid: "b3fff158beff4f7795c18a4958321d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86cab385c444e198db095959aba51b" is_pre_election: true
I20260619 14:45:19.021265 13688 leader_election.cc:288] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [CANDIDATE]: Term 1 election: Requested vote from peers b3fff158beff4f7795c18a4958321d64 (127.0.98.3:39379), ac86cab385c444e198db095959aba51b (127.0.98.1:43949)
I20260619 14:45:19.021266 13330 raft_consensus.cc:2454] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P ac86cab385c444e198db095959aba51b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3fff158beff4f7795c18a4958321d64 in term 0.
I20260619 14:45:19.021697 13540 leader_election.cc:302] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac86cab385c444e198db095959aba51b, b3fff158beff4f7795c18a4958321d64; no voters:
I20260619 14:45:19.021747 13330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfb57c5ed0543ed8bb3bb8cb71c4bf4" candidate_uuid: "b05b1b52cb4d4f5597ce0992b574c3b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86cab385c444e198db095959aba51b"
I20260619 14:45:19.021857 13330 raft_consensus.cc:3037] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P ac86cab385c444e198db095959aba51b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:45:19.022138 13470 raft_consensus.cc:2379] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3fff158beff4f7795c18a4958321d64 in current term 1: Already voted for candidate b05b1b52cb4d4f5597ce0992b574c3b4 in this term.
I20260619 14:45:19.023191 13330 raft_consensus.cc:2454] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P ac86cab385c444e198db095959aba51b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b05b1b52cb4d4f5597ce0992b574c3b4 in term 1.
I20260619 14:45:19.023402 13689 raft_consensus.cc:2781] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:45:19.023540 13689 raft_consensus.cc:482] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:45:19.023598 13689 raft_consensus.cc:3037] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:45:19.024844 13689 raft_consensus.cc:504] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fff158beff4f7795c18a4958321d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } } peers { permanent_uuid: "b05b1b52cb4d4f5597ce0992b574c3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34519 } } peers { permanent_uuid: "ac86cab385c444e198db095959aba51b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43949 } }
I20260619 14:45:19.025236 13689 leader_election.cc:288] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [CANDIDATE]: Term 1 election: Requested vote from peers b05b1b52cb4d4f5597ce0992b574c3b4 (127.0.98.2:34519), ac86cab385c444e198db095959aba51b (127.0.98.1:43949)
I20260619 14:45:19.026069 13406 leader_election.cc:302] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac86cab385c444e198db095959aba51b, b05b1b52cb4d4f5597ce0992b574c3b4; no voters:
I20260619 14:45:19.026317 13470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfb57c5ed0543ed8bb3bb8cb71c4bf4" candidate_uuid: "b3fff158beff4f7795c18a4958321d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b05b1b52cb4d4f5597ce0992b574c3b4"
I20260619 14:45:19.026484 13470 raft_consensus.cc:2379] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b3fff158beff4f7795c18a4958321d64 in current term 1: Already voted for candidate b05b1b52cb4d4f5597ce0992b574c3b4 in this term.
I20260619 14:45:19.027149 13587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfb57c5ed0543ed8bb3bb8cb71c4bf4" candidate_uuid: "b05b1b52cb4d4f5597ce0992b574c3b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3fff158beff4f7795c18a4958321d64"
I20260619 14:45:19.027340 13587 raft_consensus.cc:2379] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b05b1b52cb4d4f5597ce0992b574c3b4 in current term 1: Already voted for candidate b3fff158beff4f7795c18a4958321d64 in this term.
I20260619 14:45:19.028069 13330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfb57c5ed0543ed8bb3bb8cb71c4bf4" candidate_uuid: "b3fff158beff4f7795c18a4958321d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac86cab385c444e198db095959aba51b"
I20260619 14:45:19.028262 13330 raft_consensus.cc:2379] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P ac86cab385c444e198db095959aba51b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b3fff158beff4f7795c18a4958321d64 in current term 1: Already voted for candidate b05b1b52cb4d4f5597ce0992b574c3b4 in this term.
I20260619 14:45:19.028695 13540 leader_election.cc:302] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3fff158beff4f7795c18a4958321d64; no voters: ac86cab385c444e198db095959aba51b, b05b1b52cb4d4f5597ce0992b574c3b4
I20260619 14:45:19.028944 13689 raft_consensus.cc:2726] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 14:45:19.029131 13688 raft_consensus.cc:2781] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:45:19.029378 13688 raft_consensus.cc:686] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [term 1 LEADER]: Becoming Leader. State: Replica: b05b1b52cb4d4f5597ce0992b574c3b4, State: Running, Role: LEADER
I20260619 14:45:19.029770 13688 consensus_queue.cc:237] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fff158beff4f7795c18a4958321d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } } peers { permanent_uuid: "b05b1b52cb4d4f5597ce0992b574c3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34519 } } peers { permanent_uuid: "ac86cab385c444e198db095959aba51b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43949 } }
I20260619 14:45:19.033313 13184 catalog_manager.cc:5495] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b05b1b52cb4d4f5597ce0992b574c3b4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b05b1b52cb4d4f5597ce0992b574c3b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fff158beff4f7795c18a4958321d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b05b1b52cb4d4f5597ce0992b574c3b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac86cab385c444e198db095959aba51b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43949 } health_report { overall_health: UNKNOWN } } }
I20260619 14:45:19.112941 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:19.106982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.0.98.2:34519)
0619 14:45:19.107118 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:19.107123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:19.107140 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:45:19.107328 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:19.107332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:19.107406 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:19.107555 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:45:19.107955 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:19.108782 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:19.110109 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:19.111053 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:19.111161 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:19.111646 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:45:19.112232 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:45:19.112348 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:19.112700 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 14:45:19.112764 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:45:19.127908 13587 raft_consensus.cc:1261] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [term 1 FOLLOWER]: Refusing update from remote peer b05b1b52cb4d4f5597ce0992b574c3b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:45:19.127908 13330 raft_consensus.cc:1261] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P ac86cab385c444e198db095959aba51b [term 1 FOLLOWER]: Refusing update from remote peer b05b1b52cb4d4f5597ce0992b574c3b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:45:19.128750 13700 consensus_queue.cc:1035] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3fff158beff4f7795c18a4958321d64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:45:19.129635 13700 consensus_queue.cc:1035] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac86cab385c444e198db095959aba51b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:45:19.140939 13703 mvcc.cc:205] Tried to move back new op lower bound from 7298581787139268608 to 7298581786749755392. Current Snapshot: MvccSnapshot[applied={T|T < 7298581787139268608}]
I20260619 14:45:19.149148 13707 mvcc.cc:205] Tried to move back new op lower bound from 7298581787139268608 to 7298581786749755392. Current Snapshot: MvccSnapshot[applied={T|T < 7298581787139268608}]
I20260619 14:45:19.152118 13184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:45:19.152325 13184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:45:19.152593 13706 mvcc.cc:205] Tried to move back new op lower bound from 7298581787139268608 to 7298581786749755392. Current Snapshot: MvccSnapshot[applied={T|T < 7298581787139268608}]
I20260619 14:45:19.154918 13184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08ab7d33e6804e54b64770e789b36ca6: Sending DeleteTablet for 3 replicas of tablet 6cfb57c5ed0543ed8bb3bb8cb71c4bf4
I20260619 14:45:19.155647 13446 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:45:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43976
I20260619 14:45:19.156159 13716 tablet_replica.cc:330] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4: stopping tablet replica
I20260619 14:45:19.156116 13581 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:45:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48680
I20260619 14:45:19.156631 13716 raft_consensus.cc:2229] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:45:19.157017 13716 raft_consensus.cc:2258] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:45:19.157620 13309 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:45:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41574
I20260619 14:45:19.158627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 13223
I20260619 14:45:19.160178 13716 ts_tablet_manager.cc:1905] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:45:19.163194 13716 ts_tablet_manager.cc:1918] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:45:19.163582 13717 tablet_replica.cc:330] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64: stopping tablet replica
I20260619 14:45:19.163759 13716 log.cc:1196] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/6cfb57c5ed0543ed8bb3bb8cb71c4bf4
I20260619 14:45:19.164145 13717 raft_consensus.cc:2229] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:45:19.164680 13716 ts_tablet_manager.cc:1939] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b05b1b52cb4d4f5597ce0992b574c3b4: Deleting consensus metadata
I20260619 14:45:19.165078 13717 raft_consensus.cc:2258] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:45:19.166508 13169 catalog_manager.cc:4841] TS b05b1b52cb4d4f5597ce0992b574c3b4 (127.0.98.2:34519): tablet 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 (table test-table [id=a383d5b5ad7a41edac6571c8d2977b5d]) successfully deleted
I20260619 14:45:19.168792 13717 ts_tablet_manager.cc:1905] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:45:19.171093 13717 ts_tablet_manager.cc:1918] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:45:19.171214 13717 log.cc:1196] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/6cfb57c5ed0543ed8bb3bb8cb71c4bf4
I20260619 14:45:19.171622 13717 ts_tablet_manager.cc:1939] T 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 P b3fff158beff4f7795c18a4958321d64: Deleting consensus metadata
I20260619 14:45:19.172803 13172 catalog_manager.cc:4841] TS b3fff158beff4f7795c18a4958321d64 (127.0.98.3:39379): tablet 6cfb57c5ed0543ed8bb3bb8cb71c4bf4 (table test-table [id=a383d5b5ad7a41edac6571c8d2977b5d]) successfully deleted
W20260619 14:45:19.172923 13171 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43949 (error 108)
I20260619 14:45:19.173738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 13387
I20260619 14:45:19.174038 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:19.173732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43949 (local address 127.0.0.1:41582)
0619 14:45:19.173885 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:19.173951 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 14:45:19.174669 13171 catalog_manager.cc:4584] TS ac86cab385c444e198db095959aba51b (127.0.98.1:43949): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6cfb57c5ed0543ed8bb3bb8cb71c4bf4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43949: connect: Connection refused (error 111)
I20260619 14:45:19.179795 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 13520
I20260619 14:45:19.185793 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 13152
I20260619 14:45:19.192937 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 14:45:19.538797 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 14:45:19.539142 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 14:45:19.885347 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 14:45:19.885 UTC [12218] LOG: received smart shutdown request
2026-06-19 14:45:19.889 UTC [12218] LOG: background worker "logical replication launcher" (PID 12227) exited with exit code 1
2026-06-19 14:45:19.889 UTC [12222] LOG: shutting down
2026-06-19 14:45:20.113 UTC [12218] LOG: database system is shut down
2026-06-19T14:45:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50288 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:45:20 dist-test-slave-cr22 krb5kdc[13731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:45:20 dist-test-slave-cr22 krb5kdc[13731](info): set up 2 sockets
Jun 19 14:45:20 dist-test-slave-cr22 krb5kdc[13731](info): commencing operation
krb5kdc: starting...
W20260619 14:45:21.368330 392 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:45:21 dist-test-slave-cr22 krb5kdc[13731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:45:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:45:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:45:21.507589 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34261
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34261,127.0.98.61:42635,127.0.98.60:35397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:45:21.625180 13744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:21.625460 13744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:21.625502 13744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:21.629441 13744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:45:21.629500 13744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:21.629521 13744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:21.629539 13744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:45:21.629554 13744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:45:21.635186 13744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34261,127.0.98.61:42635,127.0.98.60:35397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:21.636466 13744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:21.637522 13744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:21.646377 13752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:21.646394 13749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:21.646382 13750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:21.646634 13744 server_base.cc:1034] running on GCE node
I20260619 14:45:21.647531 13744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:21.648717 13744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:21.649935 13744 hybrid_clock.cc:648] HybridClock initialized: now 1781880321649899 us; error 53 us; skew 500 ppm
Jun 19 14:45:21 dist-test-slave-cr22 krb5kdc[13731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:21.656834 13744 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:45:21.657919 13744 webserver.cc:460] Webserver started at http://127.0.98.62:38771/ using document root <none> and password file <none>
I20260619 14:45:21.658516 13744 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:21.658624 13744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:21.658871 13744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:45:21.661090 13744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "3850167c4cfd452e86012ac8ecff5952"
format_stamp: "Formatted at 2026-06-19 14:45:21 on dist-test-slave-cr22"
server_key: "6a7a2078055ce0b145bf44d155b3346f"
server_key_iv: "9af05226dc463aa152aea1dc596c324d"
server_key_version: "encryptionkey@0"
I20260619 14:45:21.661680 13744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "3850167c4cfd452e86012ac8ecff5952"
format_stamp: "Formatted at 2026-06-19 14:45:21 on dist-test-slave-cr22"
server_key: "6a7a2078055ce0b145bf44d155b3346f"
server_key_iv: "9af05226dc463aa152aea1dc596c324d"
server_key_version: "encryptionkey@0"
I20260619 14:45:21.665681 13744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:45:21.668287 13759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:21.669759 13744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:45:21.669934 13744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "3850167c4cfd452e86012ac8ecff5952"
format_stamp: "Formatted at 2026-06-19 14:45:21 on dist-test-slave-cr22"
server_key: "6a7a2078055ce0b145bf44d155b3346f"
server_key_iv: "9af05226dc463aa152aea1dc596c324d"
server_key_version: "encryptionkey@0"
I20260619 14:45:21.670068 13744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:21.697850 13744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:21.701846 13744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:21.702066 13744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:21.710870 13744 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34261
I20260619 14:45:21.710943 13811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34261 every 8 connection(s)
I20260619 14:45:21.712054 13744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:45:21.715085 13812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:21.715109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 13744
I20260619 14:45:21.715260 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:45:21.715627 392 external_mini_cluster.cc:1428] Setting key 40500a522f76ca9b6f956efb7f991e45
I20260619 14:45:21.719187 13812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 14:45:21 dist-test-slave-cr22 krb5kdc[13731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:45:21.736212 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:21.721836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34261 (local address 127.0.0.1:43870)
0619 14:45:21.722420 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:21.722474 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:21.723404 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:21.724247 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:21.724252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:21.724680 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:21.724948 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:21.724962 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:21.726118 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:21.726122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:21.726927 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:21.726944 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:21.727644 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:21.727648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:21.727760 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:21.728430 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:21.728459 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:21.730386 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:21.732846 (+ 2460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:21.732860 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:21.732873 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:21.733239 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:21.733664 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:21.733670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:21.733673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:21.733846 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:21.734490 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:21.734502 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:21.734703 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 14:45:21.734936 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 14:45:21.735341 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":66,"spinlock_wait_cycles":14208,"thread_start_us":167,"threads_started":1}
I20260619 14:45:21.736215 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:21.721835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43870 (local address 127.0.98.62:34261)
0619 14:45:21.722420 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:21.722439 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:21.722627 (+ 188us) server_negotiation.cc:407] Connection header received
0619 14:45:21.723828 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:21.723836 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:21.724054 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:21.724250 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:21.725114 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:21.725976 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:21.727062 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:21.727427 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:21.727661 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:21.730573 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:21.730579 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:21.730590 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:21.730613 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:21.732716 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:21.733378 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:21.733383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:21.733384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:21.733492 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:21.733955 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:21.733958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:21.733959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:21.734344 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:21.734447 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:21.734998 (+ 551us) server_negotiation.cc:299] Negotiation successful
0619 14:45:21.735242 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":482,"thread_start_us":208,"threads_started":1}
I20260619 14:45:21.738744 13812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:45:21.739636 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:21.739310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42635 (local address 127.0.0.1:54360)
0619 14:45:21.739483 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:21.739575 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 14:45:21.739845 13762 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42635: connect: Connection refused (error 111)
I20260619 14:45:21.740898 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:21.740530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42635 (local address 127.0.0.1:54372)
0619 14:45:21.740769 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:21.740839 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260619 14:45:21.741122 13812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42635: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42635: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 14:45:21.747538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42635
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34261,127.0.98.61:42635,127.0.98.60:35397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:45:21.790467 13812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } attempt: 1
I20260619 14:45:21.791595 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:21.791157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42635 (local address 127.0.0.1:54386)
0619 14:45:21.791348 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:21.791413 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 14:45:21.792479 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:21.792277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42635 (local address 127.0.0.1:54392)
0619 14:45:21.792383 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:21.792429 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 14:45:21.792702 13812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42635: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42635: connect: Connection refused (error 111)
I20260619 14:45:21.860931 13812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } attempt: 2
I20260619 14:45:21.861936 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:21.861627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42635 (local address 127.0.0.1:54396)
0619 14:45:21.861791 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:21.861857 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 14:45:21.862897 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:21.862561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42635 (local address 127.0.0.1:54412)
0619 14:45:21.862757 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:21.862817 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 14:45:21.863101 13812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42635: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42635: connect: Connection refused (error 111)
W20260619 14:45:21.867962 13819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:21.868273 13819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:21.868376 13819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:21.872648 13819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:45:21.872737 13819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:21.872784 13819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:21.872826 13819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:45:21.872859 13819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:45:21.879910 13819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34261,127.0.98.61:42635,127.0.98.60:35397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:21.881152 13819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:21.882241 13819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:21.891526 13827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:21.891528 13825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:21.891528 13824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:21.892083 13819 server_base.cc:1034] running on GCE node
I20260619 14:45:21.892587 13819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:21.893771 13819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:21.894973 13819 hybrid_clock.cc:648] HybridClock initialized: now 1781880321894937 us; error 51 us; skew 500 ppm
Jun 19 14:45:21 dist-test-slave-cr22 krb5kdc[13731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:21.898267 13819 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 14:45:21.899236 13819 webserver.cc:460] Webserver started at http://127.0.98.61:46343/ using document root <none> and password file <none>
I20260619 14:45:21.899874 13819 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:21.899972 13819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:21.900234 13819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:45:21.902328 13819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-1/data/instance:
uuid: "4c9bec84e0d84287bc077ecfa80e3477"
format_stamp: "Formatted at 2026-06-19 14:45:21 on dist-test-slave-cr22"
server_key: "7175943e6ec89bdcbe377e778bd5ff7d"
server_key_iv: "58a986e0d24a3afbf619d2c63cb0487d"
server_key_version: "encryptionkey@0"
I20260619 14:45:21.902917 13819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-1/wal/instance:
uuid: "4c9bec84e0d84287bc077ecfa80e3477"
format_stamp: "Formatted at 2026-06-19 14:45:21 on dist-test-slave-cr22"
server_key: "7175943e6ec89bdcbe377e778bd5ff7d"
server_key_iv: "58a986e0d24a3afbf619d2c63cb0487d"
server_key_version: "encryptionkey@0"
I20260619 14:45:21.906826 13819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:45:21.909525 13834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:21.910879 13819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:45:21.911134 13819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-1/wal
uuid: "4c9bec84e0d84287bc077ecfa80e3477"
format_stamp: "Formatted at 2026-06-19 14:45:21 on dist-test-slave-cr22"
server_key: "7175943e6ec89bdcbe377e778bd5ff7d"
server_key_iv: "58a986e0d24a3afbf619d2c63cb0487d"
server_key_version: "encryptionkey@0"
I20260619 14:45:21.911264 13819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:21.925485 13819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:21.928992 13819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:21.929220 13819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:21.937088 13819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42635
I20260619 14:45:21.937175 13886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42635 every 8 connection(s)
I20260619 14:45:21.938077 13819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-1/data/info.pb
I20260619 14:45:21.941059 13887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:21.944552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 13819
I20260619 14:45:21.944674 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-1/wal/instance
I20260619 14:45:21.944949 392 external_mini_cluster.cc:1428] Setting key 5b5fbe1444e2b1f6941d545da1ffd557
I20260619 14:45:21.945283 13887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260619 14:45:21.954267 13812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 14:45:21 dist-test-slave-cr22 krb5kdc[13731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 14:45:21 dist-test-slave-cr22 krb5kdc[13731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:45:21.962940 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:21.947646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43872 (local address 127.0.98.62:34261)
0619 14:45:21.947841 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:21.947848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:21.948633 (+ 785us) server_negotiation.cc:407] Connection header received
0619 14:45:21.949672 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:21.949677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:21.949752 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:21.949878 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:21.951121 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:21.952132 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:21.953446 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:21.953886 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:21.954004 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:21.957422 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:21.957426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:21.957428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:21.957457 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:21.959966 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:21.960718 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:21.960722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:21.960724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:21.960826 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:21.961388 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:21.961394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:21.961397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:21.961595 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:21.961713 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:21.962394 (+ 681us) server_negotiation.cc:299] Negotiation successful
0619 14:45:21.962773 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 14:45:21.963755 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:21.947951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34261 (local address 127.0.0.1:43872)
0619 14:45:21.948441 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:21.948494 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:21.949432 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:21.950027 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:21.950034 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:21.950375 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:21.950929 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:21.950942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:21.952301 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:21.952304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:21.953255 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:21.953266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:21.954049 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:21.954052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:21.954218 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:21.955231 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:21.955247 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:21.957273 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:21.960107 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:21.960114 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:21.960134 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:21.960589 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:21.961056 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:21.961062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:21.961067 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:21.961271 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:21.961726 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:21.961734 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:21.962092 (+ 358us) client_negotiation.cc:769] Sending connection context
0619 14:45:21.962344 (+ 252us) client_negotiation.cc:240] Negotiation successful
0619 14:45:21.962696 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":155,"threads_started":1}
I20260619 14:45:21.965258 13887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:45:21.965778 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:21.954845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42635 (local address 127.0.0.1:54420)
0619 14:45:21.954986 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:21.954998 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:21.955066 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:21.955849 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:21.955852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:21.956043 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:21.956195 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:21.956201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:21.957551 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:21.957555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:21.958371 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:21.958385 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:21.959328 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:21.959334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:21.959486 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:21.959915 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:21.959932 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:21.961309 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:21.963917 (+ 2608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:21.963922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:21.963926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:21.964179 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:21.964550 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:21.964553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:21.964555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:21.964644 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:21.965229 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:21.965232 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:21.965350 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 14:45:21.965451 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 14:45:21.965594 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 14:45:21.966831 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:21.954949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.0.98.61:42635)
0619 14:45:21.955282 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:21.955291 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:21.955311 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:45:21.955346 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:21.955350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:21.955690 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:21.955868 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:21.956307 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:21.957414 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:21.958523 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:21.959135 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:21.959316 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:21.961551 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:21.961564 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:21.961582 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:21.961614 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:21.963772 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:21.964321 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:21.964327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:21.964331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:21.964424 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:21.964780 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:21.964783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:21.964785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:21.965095 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:21.966234 (+ 1139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:21.966346 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 14:45:21.966506 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":98,"threads_started":1}
I20260619 14:45:21.969128 13812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:45:21.969947 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:21.969666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35397 (local address 127.0.0.1:39056)
0619 14:45:21.969825 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:21.969880 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 14:45:21.970798 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:21.970502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35397 (local address 127.0.0.1:39058)
0619 14:45:21.970680 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:21.970739 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260619 14:45:21.970969 13812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35397: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35397: connect: Connection refused (error 111)
Jun 19 14:45:21 dist-test-slave-cr22 krb5kdc[13731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 14:45:21.975521 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:21.965800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42635 (local address 127.0.0.1:54422)
0619 14:45:21.966119 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:21.966133 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:21.966210 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:21.966484 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:21.966487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:21.966656 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:21.966746 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:21.966752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:21.967736 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:21.967740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:21.968480 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:21.968490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:21.969583 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:21.969586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:21.969678 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:21.970160 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:21.970174 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:21.971427 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:21.973889 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:21.973892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:21.973893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:21.974089 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:21.974452 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:21.974455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:21.974457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:21.974539 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:21.974982 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:21.974984 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:21.975065 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 14:45:21.975149 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 14:45:21.975309 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
I20260619 14:45:21.975618 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:21.965869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54422 (local address 127.0.98.61:42635)
0619 14:45:21.966288 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:21.966296 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:21.966308 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:45:21.966334 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:21.966336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:21.966378 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:21.966477 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:21.966853 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:21.967600 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:21.968660 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:21.969049 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:21.969654 (+ 605us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:21.971656 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:21.971659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:21.971661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:21.971683 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:21.973726 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:21.974186 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:21.974190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:21.974193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:21.974329 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:21.974627 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:21.974629 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:21.974630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:21.974859 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:21.974938 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:21.975184 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 14:45:21.975375 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":350,"spinlock_wait_cycles":2304,"thread_start_us":137,"threads_started":1}
I20260619 14:45:21.976213 13887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:45:21.976590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35397
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34261,127.0.98.61:42635,127.0.98.60:35397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:45:21.976987 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:21.976704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35397 (local address 127.0.0.1:39072)
0619 14:45:21.976872 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:21.976938 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260619 14:45:21.977276 13838 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35397: connect: Connection refused (error 111)
I20260619 14:45:21.978212 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:21.977951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35397 (local address 127.0.0.1:39078)
0619 14:45:21.978075 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:21.978134 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 14:45:21.978370 13887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35397: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35397: connect: Connection refused (error 111)
I20260619 14:45:22.002152 13812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } attempt: 1
I20260619 14:45:22.003119 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.002766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35397 (local address 127.0.0.1:39084)
0619 14:45:22.002939 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:22.003013 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 14:45:22.003980 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.003750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35397 (local address 127.0.0.1:39086)
0619 14:45:22.003859 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:22.003922 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 14:45:22.004166 13812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35397: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35397: connect: Connection refused (error 111)
I20260619 14:45:22.027649 13887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } attempt: 1
I20260619 14:45:22.028617 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.028270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35397 (local address 127.0.0.1:39094)
0619 14:45:22.028430 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:22.028534 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 14:45:22.029408 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.029207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35397 (local address 127.0.0.1:39110)
0619 14:45:22.029309 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:22.029357 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 14:45:22.029654 13887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35397: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35397: connect: Connection refused (error 111)
I20260619 14:45:22.079402 13812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } attempt: 2
I20260619 14:45:22.080369 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.080011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35397 (local address 127.0.0.1:39126)
0619 14:45:22.080193 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:22.080267 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 14:45:22.081190 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.080938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35397 (local address 127.0.0.1:39136)
0619 14:45:22.081071 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:22.081125 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 14:45:22.081375 13812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35397: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35397: connect: Connection refused (error 111)
W20260619 14:45:22.092276 13895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:22.092680 13895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:22.092792 13895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:22.096980 13895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:45:22.097074 13895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:22.097146 13895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:22.097205 13895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:45:22.097244 13895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:45:22.097836 13887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } attempt: 2
I20260619 14:45:22.098723 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.098417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35397 (local address 127.0.0.1:39142)
0619 14:45:22.098575 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:22.098641 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 14:45:22.099511 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.099247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35397 (local address 127.0.0.1:39152)
0619 14:45:22.099353 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:22.099408 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 14:45:22.099722 13887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35397: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35397: connect: Connection refused (error 111)
I20260619 14:45:22.103111 13895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34261,127.0.98.61:42635,127.0.98.60:35397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:22.104298 13895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:22.105376 13895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:22.114249 13903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:22.114308 13895 server_base.cc:1034] running on GCE node
W20260619 14:45:22.114251 13901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:22.114251 13900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:22.115175 13895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:22.116464 13895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:22.117672 13895 hybrid_clock.cc:648] HybridClock initialized: now 1781880322117642 us; error 49 us; skew 500 ppm
Jun 19 14:45:22 dist-test-slave-cr22 krb5kdc[13731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:22.120704 13895 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 14:45:22.121662 13895 webserver.cc:460] Webserver started at http://127.0.98.60:35329/ using document root <none> and password file <none>
I20260619 14:45:22.122236 13895 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:22.122332 13895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:22.122562 13895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:45:22.124744 13895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-2/data/instance:
uuid: "b3a2fd9596db479bb6a401852dbac71f"
format_stamp: "Formatted at 2026-06-19 14:45:22 on dist-test-slave-cr22"
server_key: "e9757e8347863c712c328cd3b2f74212"
server_key_iv: "89bc25cbba3edced812ebdd52a25fbac"
server_key_version: "encryptionkey@0"
I20260619 14:45:22.125313 13895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-2/wal/instance:
uuid: "b3a2fd9596db479bb6a401852dbac71f"
format_stamp: "Formatted at 2026-06-19 14:45:22 on dist-test-slave-cr22"
server_key: "e9757e8347863c712c328cd3b2f74212"
server_key_iv: "89bc25cbba3edced812ebdd52a25fbac"
server_key_version: "encryptionkey@0"
I20260619 14:45:22.129041 13895 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:45:22.131582 13910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:22.132915 13895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:45:22.133087 13895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-2/wal
uuid: "b3a2fd9596db479bb6a401852dbac71f"
format_stamp: "Formatted at 2026-06-19 14:45:22 on dist-test-slave-cr22"
server_key: "e9757e8347863c712c328cd3b2f74212"
server_key_iv: "89bc25cbba3edced812ebdd52a25fbac"
server_key_version: "encryptionkey@0"
I20260619 14:45:22.133208 13895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:22.153177 13895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:22.156574 13895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:22.156790 13895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:22.164137 13895 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35397
I20260619 14:45:22.164167 13962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35397 every 8 connection(s)
I20260619 14:45:22.165088 13895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-2/data/info.pb
I20260619 14:45:22.168262 13963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:22.172360 13963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:45:22.173705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 13895
I20260619 14:45:22.173839 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880155260304-392-0/minicluster-data/master-2/wal/instance
I20260619 14:45:22.174139 392 external_mini_cluster.cc:1428] Setting key c35f54a96dac165b0618a6f998dd6838
I20260619 14:45:22.180579 13812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } attempt: 3
Jun 19 14:45:22 dist-test-slave-cr22 krb5kdc[13731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 14:45:22 dist-test-slave-cr22 krb5kdc[13731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 14:45:22 dist-test-slave-cr22 krb5kdc[13731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:45:22.188413 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.174541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43884 (local address 127.0.98.62:34261)
0619 14:45:22.174692 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:22.174697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:22.175702 (+ 1005us) server_negotiation.cc:407] Connection header received
0619 14:45:22.176658 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:22.176662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:22.176725 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:22.176815 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:22.177913 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.178827 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.180207 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.180499 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.180604 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:22.183845 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:22.183849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:22.183852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:22.183880 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:22.185905 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.186537 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.186542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.186543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.186636 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.187112 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.187115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.187117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.187306 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:22.187373 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:22.188124 (+ 751us) server_negotiation.cc:299] Negotiation successful
0619 14:45:22.188240 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:45:22.189432 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.175045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34261 (local address 127.0.0.1:43884)
0619 14:45:22.175535 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:22.175589 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:22.176383 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:22.176902 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:22.176909 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:22.177372 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:22.177745 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:22.177768 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.178955 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.178958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:22.180064 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:22.180074 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.180772 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.180778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:22.180984 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:22.181635 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:22.181656 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:22.183659 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:22.186027 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:22.186036 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:22.186047 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:22.186382 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:22.186761 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:22.186764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:22.186766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:22.186992 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:22.187431 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:22.187493 (+ 62us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:22.187840 (+ 347us) client_negotiation.cc:769] Sending connection context
0619 14:45:22.188068 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 14:45:22.188369 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":142,"threads_started":1}
I20260619 14:45:22.189633 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.175043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39156 (local address 127.0.98.60:35397)
0619 14:45:22.175536 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:22.175556 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:22.175589 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:45:22.176339 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:22.176356 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:22.176618 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:22.176811 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:22.177802 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.179198 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.180375 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.181377 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.182001 (+ 624us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:22.184394 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:22.184404 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:22.184419 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:22.184448 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:22.186638 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.187383 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.187391 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.187393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.187520 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.187834 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.187839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.187841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.188179 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:22.189233 (+ 1054us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:22.189337 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 14:45:22.189474 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":161,"threads_started":1}
I20260619 14:45:22.190778 13963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:45:22.190896 13887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } attempt: 3
I20260619 14:45:22.191166 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.180977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35397 (local address 127.0.0.1:39166)
0619 14:45:22.181110 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:22.181125 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:22.181196 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:22.181821 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:22.181824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:22.182053 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:22.182168 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:22.182178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.183184 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.183189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:22.183845 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:22.183853 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.184545 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.184553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:22.184658 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:22.185289 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:22.185306 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:22.186733 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:22.189356 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:22.189360 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:22.189362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:22.189622 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:22.190128 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:22.190131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:22.190133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:22.190229 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:22.190675 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:22.190679 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:22.190789 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 14:45:22.190864 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 14:45:22.191006 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 14:45:22.191843 13968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.181323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39166 (local address 127.0.98.60:35397)
0619 14:45:22.181621 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:22.181628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:22.181639 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:45:22.181664 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:22.181666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:22.181697 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:22.181807 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:22.182272 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.183056 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.183965 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.184356 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.184503 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:22.186849 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:22.186852 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:22.186855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:22.186876 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:22.189251 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.189920 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.189923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.189924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.190016 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.190328 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.190337 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.190339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.190543 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:22.191461 (+ 918us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:22.191539 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 14:45:22.191664 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":102,"threads_started":1}
I20260619 14:45:22.195391 13812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952: Bootstrap starting.
Jun 19 14:45:22 dist-test-slave-cr22 krb5kdc[13731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 14:45:22 dist-test-slave-cr22 krb5kdc[13731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:45:22.198395 13812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952: Neither blocks nor log segments found. Creating new log.
Jun 19 14:45:22 dist-test-slave-cr22 krb5kdc[13731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:45:22.199397 13812 log.cc:824] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:22.200449 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.191354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35397 (local address 127.0.0.1:39182)
0619 14:45:22.191493 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:22.191507 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:22.191594 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:22.191851 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:22.191854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:22.192061 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:22.192170 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:22.192177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.193164 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.193168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:22.193926 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:22.193935 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.194526 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.194530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:22.194630 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:22.195099 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:22.195113 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:22.196647 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:22.198652 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:22.198655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:22.198657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:22.198898 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:22.199249 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:22.199254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:22.199256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:22.199356 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:22.199876 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:22.199881 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:22.199990 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 14:45:22.200078 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 14:45:22.200257 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 14:45:22.201553 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.191367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39182 (local address 127.0.98.60:35397)
0619 14:45:22.191487 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:22.191491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:22.191549 (+ 58us) server_negotiation.cc:407] Connection header received
0619 14:45:22.191699 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:22.191702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:22.191741 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:22.191835 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:22.192289 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.193051 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.194064 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.194408 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.194841 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:22.196793 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:22.196797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:22.196800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:22.196823 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:22.198544 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.199048 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.199053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.199057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.199137 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.199506 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.199512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.199518 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.199755 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:22.200971 (+ 1216us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:22.201039 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 14:45:22.201373 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":39}
I20260619 14:45:22.201872 13812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952: No bootstrap required, opened a new log
I20260619 14:45:22.203531 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.192162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.0.98.61:42635)
0619 14:45:22.192271 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:22.192276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:22.192293 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:45:22.192329 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:22.192332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:22.192371 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:22.192453 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:22.192865 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.193609 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.194636 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.195262 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.195337 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:22.198512 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:22.198517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:22.198519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:22.198542 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:22.200795 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.201551 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.201555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.201557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.201640 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.202481 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.202487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.202489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.202785 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:22.202862 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:22.203210 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 14:45:22.203327 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:45:22.204075 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.192060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42635 (local address 127.0.0.1:54432)
0619 14:45:22.192174 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:22.192188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:22.192257 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:22.192486 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:22.192489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:22.192647 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:22.192741 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:22.192747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.193732 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.193735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:22.194530 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:22.194538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.195510 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.195517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:22.195649 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:22.196086 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:22.196100 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:22.198346 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:22.200908 (+ 2562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:22.200915 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:22.200920 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:22.201432 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:22.202243 (+ 811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:22.202245 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:22.202247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:22.202345 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:22.202938 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:22.202941 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:22.203034 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 14:45:22.203703 (+ 669us) client_negotiation.cc:240] Negotiation successful
0619 14:45:22.203833 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 14:45:22.204844 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.193223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.0.98.62:34261)
0619 14:45:22.193762 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:22.193766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:22.193777 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:45:22.193806 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:22.193809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:22.193846 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:22.193911 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:22.194727 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.195402 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.196626 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.196962 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.197054 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:22.200122 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:22.200125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:22.200127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:22.200149 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:22.202170 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.202667 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.202671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.202673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.202754 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.203045 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.203048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.203050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.203242 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:22.203399 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:22.204574 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0619 14:45:22.204707 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471}
I20260619 14:45:22.205508 13887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477: Bootstrap starting.
I20260619 14:45:22.206270 13812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3850167c4cfd452e86012ac8ecff5952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } } peers { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } } peers { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } }
I20260619 14:45:22.206535 13812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:45:22.206597 13812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3850167c4cfd452e86012ac8ecff5952, State: Initialized, Role: FOLLOWER
I20260619 14:45:22.207103 13812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3850167c4cfd452e86012ac8ecff5952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } } peers { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } } peers { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } }
I20260619 14:45:22.207630 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3850167c4cfd452e86012ac8ecff5952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } } peers { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } } peers { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } } }
I20260619 14:45:22.207764 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:45:22.208261 13887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477: Neither blocks nor log segments found. Creating new log.
I20260619 14:45:22.208490 13812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:45:22.209143 13887 log.cc:824] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:22.209599 13963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:45:22.211516 13887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477: No bootstrap required, opened a new log
Jun 19 14:45:22 dist-test-slave-cr22 krb5kdc[13731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 14:45:22 dist-test-slave-cr22 krb5kdc[13731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:45:22.215989 13887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3850167c4cfd452e86012ac8ecff5952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } } peers { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } } peers { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } }
I20260619 14:45:22.216217 13887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:45:22.216265 13887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c9bec84e0d84287bc077ecfa80e3477, State: Initialized, Role: FOLLOWER
I20260619 14:45:22.216817 13887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3850167c4cfd452e86012ac8ecff5952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } } peers { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } } peers { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } }
I20260619 14:45:22.213413 13744 master_runner.cc:186] Error getting master registration for 127.0.98.61:42635: OK, instance_id { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" instance_seqno: 1781880321934692 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:45:22.217520 13744 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:45:22.218089 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.205984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.0.98.61:42635)
0619 14:45:22.206080 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:22.206084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:22.206145 (+ 61us) server_negotiation.cc:407] Connection header received
0619 14:45:22.206277 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:22.206279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:22.206320 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:22.206396 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:22.207126 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.207907 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.208914 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.209246 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.209345 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:22.212673 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:22.212677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:22.212680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:22.212703 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:22.214705 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.215245 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.215252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.215257 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.215344 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.215686 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.215691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.215693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.215967 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:22.217711 (+ 1744us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:22.217796 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 14:45:22.217912 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:45:22.218225 13887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:45:22.221016 13982 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:45:22.221091 13982 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:45:22.221513 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:45:22.221618 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3850167c4cfd452e86012ac8ecff5952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } } peers { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } } peers { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } } }
I20260619 14:45:22.221771 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:45:22.222051 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.210301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39186 (local address 127.0.98.60:35397)
0619 14:45:22.210676 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:22.210681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:22.210691 (+ 10us) server_negotiation.cc:407] Connection header received
0619 14:45:22.210721 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:22.210725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:22.210769 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:22.210839 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:22.211375 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.212158 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.213244 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.213623 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.213731 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:22.216155 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:22.216159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:22.216161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:22.216182 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:22.217869 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.218319 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.218322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.218323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.218388 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.221127 (+ 2739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.221130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.221132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.221381 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:22.221483 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:22.221809 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 14:45:22.221935 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305}
I20260619 14:45:22.222316 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.210088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35397 (local address 127.0.0.1:39186)
0619 14:45:22.210426 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:22.210442 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:22.210523 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:22.210967 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:22.210970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:22.211142 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:22.211257 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:22.211264 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.212270 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.212276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:22.213116 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:22.213128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.213850 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.213853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:22.213962 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:22.214403 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:22.214421 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:22.216035 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:22.217989 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:22.217992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:22.217994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:22.218230 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:22.220894 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:22.220898 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:22.220900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:22.220999 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:22.221546 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:22.221549 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:22.221637 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 14:45:22.222145 (+ 508us) client_negotiation.cc:240] Negotiation successful
0619 14:45:22.222238 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265}
I20260619 14:45:22.225795 13963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f: Bootstrap starting.
I20260619 14:45:22.228526 13963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f: Neither blocks nor log segments found. Creating new log.
I20260619 14:45:22.229431 13963 log.cc:824] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:22.232393 13963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f: No bootstrap required, opened a new log
I20260619 14:45:22.230149 13819 master_runner.cc:186] Error getting master registration for 127.0.98.60:35397: OK, instance_id { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" instance_seqno: 1781880322161996 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:45:22.233628 13819 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:45:22.234079 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.223839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43902 (local address 127.0.98.62:34261)
0619 14:45:22.223955 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:22.223960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:22.224615 (+ 655us) server_negotiation.cc:407] Connection header received
0619 14:45:22.224717 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:22.224719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:22.224756 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:22.224819 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:22.225679 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.226283 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.227348 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.227682 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.227788 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:22.230414 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:22.230421 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:22.230427 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:22.230470 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:22.232214 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.232717 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.232723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.232726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.232812 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.233108 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.233112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.233113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.233321 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:22.233439 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:22.233797 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 14:45:22.233919 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:45:22.238770 13963 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3850167c4cfd452e86012ac8ecff5952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } } peers { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } } peers { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } }
I20260619 14:45:22.239054 13963 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:45:22.239126 13963 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a2fd9596db479bb6a401852dbac71f, State: Initialized, Role: FOLLOWER
W20260619 14:45:22.238996 13995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:45:22.239271 13995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:45:22.239704 13963 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3850167c4cfd452e86012ac8ecff5952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } } peers { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } } peers { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } }
I20260619 14:45:22.241218 13963 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:45:22.245136 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.224314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.0.98.61:42635)
0619 14:45:22.224418 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:22.224422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:22.234748 (+ 10326us) server_negotiation.cc:407] Connection header received
0619 14:45:22.234944 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:22.234947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:22.235007 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:22.235067 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:22.235936 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.236699 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.237944 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.238295 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.238371 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:22.240127 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:22.240131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:22.240133 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:22.240159 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:22.241810 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.242424 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.242428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.242430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.242534 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.242849 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.242854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.242858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.243108 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:22.243218 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:22.244851 (+ 1633us) server_negotiation.cc:299] Negotiation successful
0619 14:45:22.244975 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 14:45:22.248827 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:22.230091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39202 (local address 127.0.98.60:35397)
0619 14:45:22.230202 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:22.230209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:22.239777 (+ 9568us) server_negotiation.cc:407] Connection header received
0619 14:45:22.240023 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:22.240027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:22.240088 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:22.240142 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:22.240978 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.241737 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.242919 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:22.243291 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:22.243365 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:22.245187 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:22.245191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:22.245194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:22.245219 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:22.246768 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.247287 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.247290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.247292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.247371 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:22.247713 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:22.247720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:22.247722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:22.247919 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:22.248070 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:22.248527 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 14:45:22.248673 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:45:22.249683 13895 master_runner.cc:263] No leader master found from master b3a2fd9596db479bb6a401852dbac71f
I20260619 14:45:22.249763 13895 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:45:22.258858 14005 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3850167c4cfd452e86012ac8ecff5952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } } peers { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } } peers { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } } }
I20260619 14:45:22.259030 14005 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f [sys.catalog]: This master's current role is: FOLLOWER
W20260619 14:45:22.259099 14016 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:45:22.259167 14016 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:45:22.271519 13970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:45:22.271775 13970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3850167c4cfd452e86012ac8ecff5952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } } peers { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } } peers { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } }
I20260619 14:45:22.272895 13970 leader_election.cc:288] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c9bec84e0d84287bc077ecfa80e3477 (127.0.98.61:42635), b3a2fd9596db479bb6a401852dbac71f (127.0.98.60:35397)
I20260619 14:45:22.273667 13937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3850167c4cfd452e86012ac8ecff5952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a2fd9596db479bb6a401852dbac71f" is_pre_election: true
I20260619 14:45:22.274024 13937 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3850167c4cfd452e86012ac8ecff5952 in term 0.
I20260619 14:45:22.275202 13861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3850167c4cfd452e86012ac8ecff5952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9bec84e0d84287bc077ecfa80e3477" is_pre_election: true
I20260619 14:45:22.275815 13763 leader_election.cc:302] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3850167c4cfd452e86012ac8ecff5952, b3a2fd9596db479bb6a401852dbac71f; no voters:
I20260619 14:45:22.277913 13861 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3850167c4cfd452e86012ac8ecff5952 in term 0.
I20260619 14:45:22.279599 13970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:45:22.279712 13970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:45:22.279762 13970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:45:22.281193 13970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3850167c4cfd452e86012ac8ecff5952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } } peers { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } } peers { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } }
I20260619 14:45:22.281584 13970 leader_election.cc:288] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [CANDIDATE]: Term 1 election: Requested vote from peers 4c9bec84e0d84287bc077ecfa80e3477 (127.0.98.61:42635), b3a2fd9596db479bb6a401852dbac71f (127.0.98.60:35397)
I20260619 14:45:22.282284 13861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3850167c4cfd452e86012ac8ecff5952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9bec84e0d84287bc077ecfa80e3477"
I20260619 14:45:22.282433 13861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:45:22.283560 13937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3850167c4cfd452e86012ac8ecff5952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a2fd9596db479bb6a401852dbac71f"
I20260619 14:45:22.283713 13937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:45:22.285076 13937 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3850167c4cfd452e86012ac8ecff5952 in term 1.
I20260619 14:45:22.285442 13763 leader_election.cc:302] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3850167c4cfd452e86012ac8ecff5952, b3a2fd9596db479bb6a401852dbac71f; no voters:
I20260619 14:45:22.286067 13861 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3850167c4cfd452e86012ac8ecff5952 in term 1.
I20260619 14:45:22.286494 13970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:45:22.289372 13970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [term 1 LEADER]: Becoming Leader. State: Replica: 3850167c4cfd452e86012ac8ecff5952, State: Running, Role: LEADER
I20260619 14:45:22.289767 13970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3850167c4cfd452e86012ac8ecff5952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } } peers { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } } peers { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } }
I20260619 14:45:22.292734 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3850167c4cfd452e86012ac8ecff5952. Latest consensus state: current_term: 1 leader_uuid: "3850167c4cfd452e86012ac8ecff5952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3850167c4cfd452e86012ac8ecff5952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } } peers { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } } peers { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } } }
I20260619 14:45:22.292867 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [sys.catalog]: This master's current role is: LEADER
I20260619 14:45:22.293264 14020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:45:22.294207 14020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:45:22.301769 13937 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f [term 1 FOLLOWER]: Refusing update from remote peer 3850167c4cfd452e86012ac8ecff5952: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:45:22.304467 14018 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:45:22.309007 14005 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3850167c4cfd452e86012ac8ecff5952. Latest consensus state: current_term: 1 leader_uuid: "3850167c4cfd452e86012ac8ecff5952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3850167c4cfd452e86012ac8ecff5952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } } peers { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } } peers { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } } }
I20260619 14:45:22.309150 14005 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:45:22.310165 13861 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477 [term 1 FOLLOWER]: Refusing update from remote peer 3850167c4cfd452e86012ac8ecff5952: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:45:22.311014 13970 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:45:22.317260 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3850167c4cfd452e86012ac8ecff5952. Latest consensus state: current_term: 1 leader_uuid: "3850167c4cfd452e86012ac8ecff5952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3850167c4cfd452e86012ac8ecff5952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } } peers { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } } peers { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } } }
I20260619 14:45:22.317394 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:45:22.324591 14018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3850167c4cfd452e86012ac8ecff5952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3850167c4cfd452e86012ac8ecff5952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } } peers { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } } peers { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } } }
I20260619 14:45:22.324746 14018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [sys.catalog]: This master's current role is: LEADER
I20260619 14:45:22.325840 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3850167c4cfd452e86012ac8ecff5952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3850167c4cfd452e86012ac8ecff5952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } } peers { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } } peers { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } } }
I20260619 14:45:22.325966 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:45:22.326354 14021 mvcc.cc:205] Tried to move back new op lower bound from 7298581800138153984 to 7298581800102572032. Current Snapshot: MvccSnapshot[applied={T|T < 7298581800138153984}]
I20260619 14:45:22.330343 14024 mvcc.cc:205] Tried to move back new op lower bound from 7298581800138153984 to 7298581800102572032. Current Snapshot: MvccSnapshot[applied={T|T < 7298581800138153984}]
I20260619 14:45:22.331251 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3850167c4cfd452e86012ac8ecff5952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3850167c4cfd452e86012ac8ecff5952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } } peers { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } } peers { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } } }
I20260619 14:45:22.331406 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952 [sys.catalog]: This master's current role is: LEADER
I20260619 14:45:22.332638 14005 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3850167c4cfd452e86012ac8ecff5952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3850167c4cfd452e86012ac8ecff5952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34261 } } peers { permanent_uuid: "4c9bec84e0d84287bc077ecfa80e3477" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42635 } } peers { permanent_uuid: "b3a2fd9596db479bb6a401852dbac71f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35397 } } }
I20260619 14:45:22.332765 14005 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:45:22.336500 14022 mvcc.cc:205] Tried to move back new op lower bound from 7298581800138153984 to 7298581800102572032. Current Snapshot: MvccSnapshot[applied={T|T < 7298581800138153984}]
I20260619 14:45:22.338117 14020 catalog_manager.cc:1347] Generated new cluster ID: f82aa4412a804aa99f5f37ba2598706f
I20260619 14:45:22.338187 14020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:45:22.372179 14020 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:45:22.373602 14020 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:45:22.405263 14020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3850167c4cfd452e86012ac8ecff5952: Generated new TSK 0
I20260619 14:45:22.406167 14020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:45:23.240585 13995 catalog_manager.cc:1259] Loaded cluster ID: f82aa4412a804aa99f5f37ba2598706f
I20260619 14:45:23.240666 13995 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477: loading cluster ID for follower catalog manager: success
I20260619 14:45:23.242903 13995 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477: acquiring CA information for follower catalog manager: success
I20260619 14:45:23.244217 13995 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4c9bec84e0d84287bc077ecfa80e3477: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:45:23.260553 14016 catalog_manager.cc:1259] Loaded cluster ID: f82aa4412a804aa99f5f37ba2598706f
I20260619 14:45:23.260648 14016 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f: loading cluster ID for follower catalog manager: success
I20260619 14:45:23.262286 14016 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f: acquiring CA information for follower catalog manager: success
I20260619 14:45:23.263423 14016 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b3a2fd9596db479bb6a401852dbac71f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:45:23.475626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 13744
I20260619 14:45:23.482765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 13819
I20260619 14:45:23.489308 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 13895
2026-06-19T14:45:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3245 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:45:23 dist-test-slave-cr22 krb5kdc[14036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:45:23 dist-test-slave-cr22 krb5kdc[14036](info): set up 2 sockets
Jun 19 14:45:23 dist-test-slave-cr22 krb5kdc[14036](info): commencing operation
krb5kdc: starting...
W20260619 14:45:24.575867 392 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:45:24 dist-test-slave-cr22 krb5kdc[14036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:45:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:45:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:45:24.711764 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46765
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:45:24.826691 14049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:24.826978 14049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:24.827020 14049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:24.831001 14049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:45:24.831138 14049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:24.831162 14049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:24.831179 14049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:45:24.831194 14049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:45:24.836603 14049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:24.837651 14049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:24.838608 14049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:24.847937 14057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:24.847923 14055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:24.847923 14054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:24.848315 14049 server_base.cc:1034] running on GCE node
I20260619 14:45:24.848909 14049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:24.850147 14049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:24.851375 14049 hybrid_clock.cc:648] HybridClock initialized: now 1781880324851345 us; error 48 us; skew 500 ppm
Jun 19 14:45:24 dist-test-slave-cr22 krb5kdc[14036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:24.854413 14049 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:45:24.855402 14049 webserver.cc:460] Webserver started at http://127.0.98.62:46141/ using document root <none> and password file <none>
I20260619 14:45:24.856081 14049 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:24.856175 14049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:24.856424 14049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:45:24.858479 14049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "2095abd6a51a4154ac8f519be74a2681"
format_stamp: "Formatted at 2026-06-19 14:45:24 on dist-test-slave-cr22"
server_key: "1fa1a2cd240828ed5c6e502cec089f52"
server_key_iv: "93e71f43d897f21f6fe7916e81e6e6ea"
server_key_version: "encryptionkey@0"
I20260619 14:45:24.859056 14049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "2095abd6a51a4154ac8f519be74a2681"
format_stamp: "Formatted at 2026-06-19 14:45:24 on dist-test-slave-cr22"
server_key: "1fa1a2cd240828ed5c6e502cec089f52"
server_key_iv: "93e71f43d897f21f6fe7916e81e6e6ea"
server_key_version: "encryptionkey@0"
I20260619 14:45:24.862879 14049 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260619 14:45:24.865424 14064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:24.866824 14049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:45:24.866982 14049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "2095abd6a51a4154ac8f519be74a2681"
format_stamp: "Formatted at 2026-06-19 14:45:24 on dist-test-slave-cr22"
server_key: "1fa1a2cd240828ed5c6e502cec089f52"
server_key_iv: "93e71f43d897f21f6fe7916e81e6e6ea"
server_key_version: "encryptionkey@0"
I20260619 14:45:24.867110 14049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:24.878463 14049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:24.882103 14049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:24.882332 14049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:24.890131 14049 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46765
I20260619 14:45:24.890178 14116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46765 every 8 connection(s)
I20260619 14:45:24.891099 14049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:45:24.894069 14117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:24.899724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 14049
I20260619 14:45:24.899830 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:45:24.900168 392 external_mini_cluster.cc:1428] Setting key 358b88e70e2202c776447a06c622b578
I20260619 14:45:24.901080 14117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681: Bootstrap starting.
I20260619 14:45:24.904246 14117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681: Neither blocks nor log segments found. Creating new log.
I20260619 14:45:24.905227 14117 log.cc:824] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:24.907522 14117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681: No bootstrap required, opened a new log
Jun 19 14:45:24 dist-test-slave-cr22 krb5kdc[14036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:45:24.912566 14117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2095abd6a51a4154ac8f519be74a2681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46765 } }
I20260619 14:45:24.912979 14117 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:45:24.913062 14117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2095abd6a51a4154ac8f519be74a2681, State: Initialized, Role: FOLLOWER
I20260619 14:45:24.913605 14117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2095abd6a51a4154ac8f519be74a2681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46765 } }
I20260619 14:45:24.913794 14117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:45:24.913859 14117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:45:24.914041 14117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:45:24.915421 14117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2095abd6a51a4154ac8f519be74a2681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46765 } }
I20260619 14:45:24.915858 14117 leader_election.cc:302] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2095abd6a51a4154ac8f519be74a2681; no voters:
I20260619 14:45:24.916307 14117 leader_election.cc:288] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:45:24.916558 14122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:45:24.917604 14120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:24.901732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60308 (local address 127.0.98.62:46765)
0619 14:45:24.902207 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:24.902227 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:24.902274 (+ 47us) server_negotiation.cc:407] Connection header received
0619 14:45:24.903271 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:24.903296 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:24.903801 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:24.904201 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:24.905021 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:24.906540 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:24.907500 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:24.908246 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:24.908408 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:24.911495 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:24.911505 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:24.911526 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:24.911561 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:24.913833 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:24.914420 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:24.914427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:24.914434 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:24.914617 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:24.914917 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:24.914920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:24.914922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:24.915558 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:24.915799 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:24.916095 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 14:45:24.916450 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":135,"threads_started":1}
I20260619 14:45:24.917699 14117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:45:24.916797 14122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681 [term 1 LEADER]: Becoming Leader. State: Replica: 2095abd6a51a4154ac8f519be74a2681, State: Running, Role: LEADER
I20260619 14:45:24.919667 14122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2095abd6a51a4154ac8f519be74a2681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46765 } }
I20260619 14:45:24.920876 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:45:24.921625 14123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2095abd6a51a4154ac8f519be74a2681. Latest consensus state: current_term: 1 leader_uuid: "2095abd6a51a4154ac8f519be74a2681" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2095abd6a51a4154ac8f519be74a2681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46765 } } }
I20260619 14:45:24.921761 14123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681 [sys.catalog]: This master's current role is: LEADER
I20260619 14:45:24.922209 14124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2095abd6a51a4154ac8f519be74a2681" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2095abd6a51a4154ac8f519be74a2681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46765 } } }
I20260619 14:45:24.922091 14137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:45:24.922303 14124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681 [sys.catalog]: This master's current role is: LEADER
I20260619 14:45:24.925024 14137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:45:24.931376 14137 catalog_manager.cc:1347] Generated new cluster ID: a6518a5edfe94bc58e5b87b5045fc2e5
I20260619 14:45:24.931521 14137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:45:24.952075 14137 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:45:24.953171 14137 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:45:24.961014 14137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2095abd6a51a4154ac8f519be74a2681: Generated new TSK 0
I20260619 14:45:24.961831 14137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:45:24.977442 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 14049
2026-06-19T14:45:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1485 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:45:25 dist-test-slave-cr22 krb5kdc[14144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:45:25 dist-test-slave-cr22 krb5kdc[14144](info): set up 2 sockets
Jun 19 14:45:25 dist-test-slave-cr22 krb5kdc[14144](info): commencing operation
krb5kdc: starting...
W20260619 14:45:26.071834 392 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:45:26 dist-test-slave-cr22 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:45:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:45:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:45:26.204650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44835
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:45:26.326287 14157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:26.326629 14157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:26.326707 14157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:26.330866 14157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:45:26.330976 14157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:26.331014 14157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:26.331067 14157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:45:26.331104 14157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:45:26.337198 14157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:26.338362 14157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:26.339397 14157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:26.348441 14163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:26.348424 14162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:26.348606 14157 server_base.cc:1034] running on GCE node
W20260619 14:45:26.348426 14165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:26.349416 14157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:26.350711 14157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:26.351918 14157 hybrid_clock.cc:648] HybridClock initialized: now 1781880326351898 us; error 32 us; skew 500 ppm
Jun 19 14:45:26 dist-test-slave-cr22 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:26.355373 14157 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:45:26.356411 14157 webserver.cc:460] Webserver started at http://127.0.98.62:34857/ using document root <none> and password file <none>
I20260619 14:45:26.357064 14157 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:26.357164 14157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:26.357414 14157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:45:26.359742 14157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4"
format_stamp: "Formatted at 2026-06-19 14:45:26 on dist-test-slave-cr22"
server_key: "e4c43d0fa6669418a4004d56764b0c3c"
server_key_iv: "8cdec2feda2c2b5e376ae19ca98c0583"
server_key_version: "encryptionkey@0"
I20260619 14:45:26.360375 14157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4"
format_stamp: "Formatted at 2026-06-19 14:45:26 on dist-test-slave-cr22"
server_key: "e4c43d0fa6669418a4004d56764b0c3c"
server_key_iv: "8cdec2feda2c2b5e376ae19ca98c0583"
server_key_version: "encryptionkey@0"
I20260619 14:45:26.364480 14157 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:45:26.367823 14172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:26.369247 14157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:45:26.369438 14157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4"
format_stamp: "Formatted at 2026-06-19 14:45:26 on dist-test-slave-cr22"
server_key: "e4c43d0fa6669418a4004d56764b0c3c"
server_key_iv: "8cdec2feda2c2b5e376ae19ca98c0583"
server_key_version: "encryptionkey@0"
I20260619 14:45:26.369572 14157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:26.387017 14157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:26.390483 14157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:26.390889 14157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:26.398826 14157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44835
I20260619 14:45:26.399147 14224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44835 every 8 connection(s)
I20260619 14:45:26.399996 14157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:45:26.401693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 14157
I20260619 14:45:26.401803 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:45:26.402117 392 external_mini_cluster.cc:1428] Setting key ceee17258c4cbe328e2a677c5c612616
I20260619 14:45:26.403137 14225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:26.409727 14225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4: Bootstrap starting.
Jun 19 14:45:26 dist-test-slave-cr22 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:45:26.412458 14225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4: Neither blocks nor log segments found. Creating new log.
I20260619 14:45:26.413194 14225 log.cc:824] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:26.415251 14225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4: No bootstrap required, opened a new log
I20260619 14:45:26.417289 14228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:26.403725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.0.98.62:44835)
0619 14:45:26.404156 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:26.404173 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:26.404209 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:45:26.404900 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:26.404916 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:26.405254 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:26.405542 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:26.406435 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:26.407667 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:26.408845 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:26.409321 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:26.409505 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:26.412497 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:26.412502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:26.412516 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:26.412545 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:26.414469 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:26.414938 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:26.414943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:26.414947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:26.415052 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:26.415349 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:26.415355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:26.415359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:26.415811 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:26.416007 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:26.416285 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 14:45:26.416590 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":139,"threads_started":1}
I20260619 14:45:26.419787 14225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44835 } }
I20260619 14:45:26.420046 14225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:45:26.420138 14225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b8c0ed4ae4a40be8b220e575bc0f4f4, State: Initialized, Role: FOLLOWER
I20260619 14:45:26.420583 14225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44835 } }
I20260619 14:45:26.420746 14225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:45:26.420826 14225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:45:26.420958 14225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:45:26.422008 14225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44835 } }
I20260619 14:45:26.422456 14225 leader_election.cc:302] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b8c0ed4ae4a40be8b220e575bc0f4f4; no voters:
I20260619 14:45:26.422847 14225 leader_election.cc:288] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:45:26.422971 14230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:45:26.423214 14230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 1 LEADER]: Becoming Leader. State: Replica: 9b8c0ed4ae4a40be8b220e575bc0f4f4, State: Running, Role: LEADER
I20260619 14:45:26.423590 14230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44835 } }
I20260619 14:45:26.424275 14225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:45:26.425410 14231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44835 } } }
I20260619 14:45:26.425529 14231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [sys.catalog]: This master's current role is: LEADER
I20260619 14:45:26.425784 14232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b8c0ed4ae4a40be8b220e575bc0f4f4. Latest consensus state: current_term: 1 leader_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44835 } } }
I20260619 14:45:26.425904 14232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [sys.catalog]: This master's current role is: LEADER
I20260619 14:45:26.426021 14239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:45:26.429497 14239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:45:26.435286 14239 catalog_manager.cc:1347] Generated new cluster ID: f03ea41ece344956a7a6e1593373d64b
I20260619 14:45:26.435364 14239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:45:26.442356 14239 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:45:26.443384 14239 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:45:26.452888 14239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4: Generated new TSK 0
I20260619 14:45:26.453616 14239 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:45:26.487119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44835
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:45:26.604866 14251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:26.605113 14251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:26.605158 14251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:26.609194 14251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:26.609254 14251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:26.609354 14251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:45:26.614666 14251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:26.615942 14251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:26.617177 14251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:26.628041 14259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:26.628144 14257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:26.628041 14256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:26.628199 14251 server_base.cc:1034] running on GCE node
I20260619 14:45:26.628834 14251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:26.629429 14251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:26.630604 14251 hybrid_clock.cc:648] HybridClock initialized: now 1781880326630581 us; error 33 us; skew 500 ppm
Jun 19 14:45:26 dist-test-slave-cr22 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:26.633764 14251 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:45:26.634657 14251 webserver.cc:460] Webserver started at http://127.0.98.1:39045/ using document root <none> and password file <none>
I20260619 14:45:26.635252 14251 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:26.635344 14251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:26.635591 14251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:45:26.637645 14251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "66d205adbdb84e52a264ef7ce665a6ff"
format_stamp: "Formatted at 2026-06-19 14:45:26 on dist-test-slave-cr22"
server_key: "d823822b4f050fc225970969263199e0"
server_key_iv: "9fc7244a758ba683f5d955360dd51d7c"
server_key_version: "encryptionkey@0"
I20260619 14:45:26.638211 14251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "66d205adbdb84e52a264ef7ce665a6ff"
format_stamp: "Formatted at 2026-06-19 14:45:26 on dist-test-slave-cr22"
server_key: "d823822b4f050fc225970969263199e0"
server_key_iv: "9fc7244a758ba683f5d955360dd51d7c"
server_key_version: "encryptionkey@0"
I20260619 14:45:26.641956 14251 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:45:26.644683 14266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:26.646049 14251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:45:26.646245 14251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "66d205adbdb84e52a264ef7ce665a6ff"
format_stamp: "Formatted at 2026-06-19 14:45:26 on dist-test-slave-cr22"
server_key: "d823822b4f050fc225970969263199e0"
server_key_iv: "9fc7244a758ba683f5d955360dd51d7c"
server_key_version: "encryptionkey@0"
I20260619 14:45:26.646389 14251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:26.663391 14251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:26.666755 14251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:26.666909 14251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:26.667510 14251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:45:26.668299 14251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:45:26.668345 14251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:26.668385 14251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:45:26.668409 14251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:26.677623 14251 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38823
I20260619 14:45:26.677655 14379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38823 every 8 connection(s)
I20260619 14:45:26.678537 14251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:45:26.684648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 14251
I20260619 14:45:26.684772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:45:26.685079 392 external_mini_cluster.cc:1428] Setting key f209a801652f25e80fbd23430c1bb3ca
Jun 19 14:45:26 dist-test-slave-cr22 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:45:26.693712 14228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:26.680427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38131 (local address 127.0.98.62:44835)
0619 14:45:26.680581 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:26.680584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:26.681391 (+ 807us) server_negotiation.cc:407] Connection header received
0619 14:45:26.682705 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:26.682709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:26.682765 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:26.682845 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:26.684004 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:26.684707 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:26.686071 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:26.686427 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:26.686538 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:26.689649 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:26.689656 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:26.689662 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:26.689696 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:26.691487 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:26.691999 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:26.692005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:26.692010 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:26.692111 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:26.692443 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:26.692449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:26.692453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:26.692736 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:26.692851 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:26.693363 (+ 512us) server_negotiation.cc:299] Negotiation successful
0619 14:45:26.693504 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:45:26.694505 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:26.680809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:38131)
0619 14:45:26.681243 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:26.681285 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:26.682495 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:26.683008 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:26.683015 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:26.683369 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:26.683858 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:26.683870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:26.684874 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:26.684883 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:26.685951 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:26.685960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:26.686606 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:26.686611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:26.686796 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:26.687427 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:26.687468 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:26.689415 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:26.691617 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:26.691622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:26.691635 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:26.691891 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:26.692215 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:26.692217 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:26.692218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:26.692349 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:26.692843 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:26.692847 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:26.693111 (+ 264us) client_negotiation.cc:769] Sending connection context
0619 14:45:26.693274 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 14:45:26.693528 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:45:26.695938 14380 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44835
I20260619 14:45:26.696195 14380 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:26.696728 14380 heartbeater.cc:511] Master 127.0.98.62:44835 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:45:26.698503 14189 ts_manager.cc:194] Registered new tserver with Master: 66d205adbdb84e52a264ef7ce665a6ff (127.0.98.1:38823)
I20260619 14:45:26.700124 14189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:45:26.716302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44835
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:45:26.830703 14385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:26.830991 14385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:26.831037 14385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:26.835237 14385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:26.835297 14385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:26.835356 14385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:45:26.840898 14385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:26.841982 14385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:26.843053 14385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:26.852870 14390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:26.852909 14391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:26.853091 14393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:26.853370 14385 server_base.cc:1034] running on GCE node
I20260619 14:45:26.853827 14385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:26.854547 14385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:26.855739 14385 hybrid_clock.cc:648] HybridClock initialized: now 1781880326855717 us; error 34 us; skew 500 ppm
Jun 19 14:45:26 dist-test-slave-cr22 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:26.858870 14385 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:45:26.859869 14385 webserver.cc:460] Webserver started at http://127.0.98.2:35891/ using document root <none> and password file <none>
I20260619 14:45:26.860442 14385 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:26.860504 14385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:26.860746 14385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:45:26.862929 14385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "1556caff2fbf42e698ccc4f464748fdc"
format_stamp: "Formatted at 2026-06-19 14:45:26 on dist-test-slave-cr22"
server_key: "bf438cc2aac8fea2d1f056f098a08f0f"
server_key_iv: "bc09a8c7d605a910eba123457dd387ca"
server_key_version: "encryptionkey@0"
I20260619 14:45:26.863593 14385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1556caff2fbf42e698ccc4f464748fdc"
format_stamp: "Formatted at 2026-06-19 14:45:26 on dist-test-slave-cr22"
server_key: "bf438cc2aac8fea2d1f056f098a08f0f"
server_key_iv: "bc09a8c7d605a910eba123457dd387ca"
server_key_version: "encryptionkey@0"
I20260619 14:45:26.867722 14385 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:45:26.870318 14400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:26.872049 14385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:45:26.872187 14385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "1556caff2fbf42e698ccc4f464748fdc"
format_stamp: "Formatted at 2026-06-19 14:45:26 on dist-test-slave-cr22"
server_key: "bf438cc2aac8fea2d1f056f098a08f0f"
server_key_iv: "bc09a8c7d605a910eba123457dd387ca"
server_key_version: "encryptionkey@0"
I20260619 14:45:26.872331 14385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:26.883810 14385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:26.887626 14385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:26.887846 14385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:26.888439 14385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:45:26.889317 14385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:45:26.889390 14385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:26.889462 14385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:45:26.889514 14385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:26.898363 14385 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33265
I20260619 14:45:26.898418 14513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33265 every 8 connection(s)
I20260619 14:45:26.899348 14385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:45:26.903281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 14385
I20260619 14:45:26.903469 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:45:26.903844 392 external_mini_cluster.cc:1428] Setting key 9569a6e880e2d488fbda7cdab28aa525
Jun 19 14:45:26 dist-test-slave-cr22 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:45:26.915526 14228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:26.901236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39943 (local address 127.0.98.62:44835)
0619 14:45:26.901396 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:26.901402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:26.902231 (+ 829us) server_negotiation.cc:407] Connection header received
0619 14:45:26.903715 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:26.903719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:26.903783 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:26.903887 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:26.905401 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:26.906253 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:26.908081 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:26.908302 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:26.908390 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:26.911359 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:26.911362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:26.911364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:26.911389 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:26.913072 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:26.913689 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:26.913694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:26.913696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:26.913812 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:26.914197 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:26.914201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:26.914202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:26.914407 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:26.914517 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:26.915199 (+ 682us) server_negotiation.cc:299] Negotiation successful
0619 14:45:26.915332 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:45:26.916486 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:26.901616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:39943)
0619 14:45:26.902067 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:26.902111 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:26.903375 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:26.904135 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:26.904142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:26.904629 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:26.905157 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:26.905178 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:26.906420 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:26.906423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:26.907942 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:26.907955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:26.908472 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:26.908478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:26.908717 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:26.909387 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:26.909415 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:26.911167 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:26.913231 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:26.913238 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:26.913255 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:26.913537 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:26.913912 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:26.913915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:26.913917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:26.914087 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:26.914528 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:26.914536 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:26.914930 (+ 394us) client_negotiation.cc:769] Sending connection context
0619 14:45:26.915154 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 14:45:26.915527 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:45:26.917896 14514 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44835
I20260619 14:45:26.918179 14514 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:26.918728 14514 heartbeater.cc:511] Master 127.0.98.62:44835 requested a full tablet report, sending...
I20260619 14:45:26.919929 14189 ts_manager.cc:194] Registered new tserver with Master: 1556caff2fbf42e698ccc4f464748fdc (127.0.98.2:33265)
I20260619 14:45:26.920615 14189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:45:26.935328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44835
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:45:27.047559 14519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:27.047869 14519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:27.047950 14519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:27.053056 14519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:27.053157 14519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:27.053251 14519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:45:27.058835 14519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:27.060196 14519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:27.061518 14519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:27.071786 14527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:27.071787 14524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:27.071847 14519 server_base.cc:1034] running on GCE node
W20260619 14:45:27.071787 14525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:27.072561 14519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:27.073226 14519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:27.074446 14519 hybrid_clock.cc:648] HybridClock initialized: now 1781880327074375 us; error 90 us; skew 500 ppm
Jun 19 14:45:27 dist-test-slave-cr22 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:27.077495 14519 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:45:27.078459 14519 webserver.cc:460] Webserver started at http://127.0.98.3:40111/ using document root <none> and password file <none>
I20260619 14:45:27.079075 14519 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:27.079172 14519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:27.079416 14519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:45:27.081710 14519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "4ce60de966ab4295ae6d6d43f9db4b5e"
format_stamp: "Formatted at 2026-06-19 14:45:27 on dist-test-slave-cr22"
server_key: "7d935537a5f474b8c18bc4d31abc6176"
server_key_iv: "60efcef9ff761cb5665033f034da0184"
server_key_version: "encryptionkey@0"
I20260619 14:45:27.082329 14519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4ce60de966ab4295ae6d6d43f9db4b5e"
format_stamp: "Formatted at 2026-06-19 14:45:27 on dist-test-slave-cr22"
server_key: "7d935537a5f474b8c18bc4d31abc6176"
server_key_iv: "60efcef9ff761cb5665033f034da0184"
server_key_version: "encryptionkey@0"
I20260619 14:45:27.086251 14519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:45:27.089047 14534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:27.090487 14519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:45:27.090657 14519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "4ce60de966ab4295ae6d6d43f9db4b5e"
format_stamp: "Formatted at 2026-06-19 14:45:27 on dist-test-slave-cr22"
server_key: "7d935537a5f474b8c18bc4d31abc6176"
server_key_iv: "60efcef9ff761cb5665033f034da0184"
server_key_version: "encryptionkey@0"
I20260619 14:45:27.090773 14519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:27.108882 14519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:27.112324 14519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:27.112486 14519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:27.113166 14519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:45:27.114009 14519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:45:27.114055 14519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:27.114127 14519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:45:27.114169 14519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:27.123644 14519 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43309
I20260619 14:45:27.123667 14647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43309 every 8 connection(s)
I20260619 14:45:27.124637 14519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:45:27.132759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 14519
I20260619 14:45:27.132905 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:45:27.133278 392 external_mini_cluster.cc:1428] Setting key 57b97f1d8fde5e92eba1eef930964b5c
Jun 19 14:45:27 dist-test-slave-cr22 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:45:27.138811 14228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.126684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37317 (local address 127.0.98.62:44835)
0619 14:45:27.126840 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:27.126845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:27.127752 (+ 907us) server_negotiation.cc:407] Connection header received
0619 14:45:27.129046 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:27.129048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:27.129097 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:27.129183 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:27.130306 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.130924 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.132155 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.132370 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.132597 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.135333 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:27.135336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:27.135338 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:27.135363 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:27.136797 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:27.137310 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:27.137315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:27.137317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:27.137400 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:27.137738 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:27.137743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:27.137745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:27.137928 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:27.138036 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:27.138523 (+ 487us) server_negotiation.cc:299] Negotiation successful
0619 14:45:27.138645 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:45:27.139688 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.127124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:37317)
0619 14:45:27.127609 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:27.127651 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:27.128826 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:27.129336 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:27.129342 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:27.129688 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:27.130155 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:27.130167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.131043 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.131046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:27.132014 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:27.132023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.132608 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.132611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:27.132745 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.133443 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:27.133459 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:27.135157 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:27.136918 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:27.136924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:27.136939 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:27.137203 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:27.137501 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:27.137505 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:27.137507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:27.137642 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:27.138032 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:27.138038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:27.138292 (+ 254us) client_negotiation.cc:769] Sending connection context
0619 14:45:27.138480 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 14:45:27.138804 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":114,"threads_started":1}
I20260619 14:45:27.140852 14648 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44835
I20260619 14:45:27.141104 14648 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:27.141603 14648 heartbeater.cc:511] Master 127.0.98.62:44835 requested a full tablet report, sending...
I20260619 14:45:27.142621 14189 ts_manager.cc:194] Registered new tserver with Master: 4ce60de966ab4295ae6d6d43f9db4b5e (127.0.98.3:43309)
I20260619 14:45:27.143213 14189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37317
I20260619 14:45:27.147228 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:45:27.156961 14228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.149356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60214 (local address 127.0.98.62:44835)
0619 14:45:27.149541 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:27.149547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:27.149796 (+ 249us) server_negotiation.cc:407] Connection header received
0619 14:45:27.149934 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:27.149937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:27.149981 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:27.150078 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:27.150927 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.151501 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.152345 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.152582 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.152693 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.154031 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:27.154034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:27.154037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:27.154070 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:27.155222 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:27.155697 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:27.155701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:27.155703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:27.155789 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:27.156064 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:27.156068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:27.156069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:27.156288 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:27.156440 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:27.156688 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 14:45:27.156803 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 14:45:27.159971 14189 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:45:27.162948 14189 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:45:27.177932 14662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.171670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33265 (local address 127.0.0.1:57882)
0619 14:45:27.172644 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:27.172698 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:27.172845 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:27.173718 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:27.173728 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:27.173760 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:27.173923 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:27.173931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.175042 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.175046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:27.176300 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:27.176310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.177404 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.177408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:27.177498 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.177526 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 14:45:27.177647 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 14:45:27.177773 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"spinlock_wait_cycles":12288,"thread_start_us":98,"threads_started":1}
I20260619 14:45:27.177942 14661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.171887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57882 (local address 127.0.98.2:33265)
0619 14:45:27.173197 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:27.173210 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:27.173238 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:45:27.173322 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:27.173328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:27.173543 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:27.173704 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:27.174055 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.174908 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.176463 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.177070 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.177183 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.177334 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:27.177680 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 14:45:27.177774 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1210,"thread_start_us":116,"threads_started":1}
I20260619 14:45:27.178658 14659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.171886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38823 (local address 127.0.0.1:41220)
0619 14:45:27.172646 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:27.172698 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:27.172845 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:27.173837 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:27.173840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:27.173855 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:27.173931 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:27.173936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.175756 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.175760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:27.176883 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:27.176892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.178243 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.178247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:27.178352 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.178385 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 14:45:27.178437 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 14:45:27.178509 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":18,"spinlock_wait_cycles":16512,"thread_start_us":121,"threads_started":1}
I20260619 14:45:27.179515 14663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.172420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41220 (local address 127.0.98.1:38823)
0619 14:45:27.173275 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:27.173293 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:27.173319 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:45:27.173453 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:27.173461 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:27.173676 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:27.174124 (+ 448us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:27.174298 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.175589 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.177024 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.177929 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.178853 (+ 924us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.179178 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:27.179235 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 14:45:27.179328 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":92,"threads_started":1}
I20260619 14:45:27.180209 14660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.172123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43309 (local address 127.0.0.1:57792)
0619 14:45:27.173074 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:27.173088 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:27.173173 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:27.173896 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:27.173898 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:27.173913 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:27.174018 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:27.174023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.175756 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.175760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:27.178275 (+ 2515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:27.178299 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.179803 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.179806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:27.179878 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.179896 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:45:27.180000 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 14:45:27.180075 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":26,"thread_start_us":102,"threads_started":1}
I20260619 14:45:27.180280 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.172545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57792 (local address 127.0.98.3:43309)
0619 14:45:27.173048 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:27.173062 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:27.173289 (+ 227us) server_negotiation.cc:407] Connection header received
0619 14:45:27.173414 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:27.173419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:27.173559 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:27.173704 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:27.174147 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.175603 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.178432 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.179303 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.179412 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.179727 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:27.180040 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 14:45:27.180136 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":100,"threads_started":1}
I20260619 14:45:27.181090 14448 tablet_service.cc:1467] Processing CreateTablet for tablet cb63da61480b4b9794465db0baa84d32 (DEFAULT_TABLE table=test-table [id=a1b4318712ee4bd08a05ee031074b0de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:45:27.182276 14448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb63da61480b4b9794465db0baa84d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:27.182981 14582 tablet_service.cc:1467] Processing CreateTablet for tablet cb63da61480b4b9794465db0baa84d32 (DEFAULT_TABLE table=test-table [id=a1b4318712ee4bd08a05ee031074b0de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:45:27.184114 14582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb63da61480b4b9794465db0baa84d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:27.185498 14314 tablet_service.cc:1467] Processing CreateTablet for tablet cb63da61480b4b9794465db0baa84d32 (DEFAULT_TABLE table=test-table [id=a1b4318712ee4bd08a05ee031074b0de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:45:27.186297 14314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb63da61480b4b9794465db0baa84d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:27.189978 14665 tablet_bootstrap.cc:492] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc: Bootstrap starting.
I20260619 14:45:27.191334 14666 tablet_bootstrap.cc:492] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e: Bootstrap starting.
I20260619 14:45:27.192390 14667 tablet_bootstrap.cc:492] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff: Bootstrap starting.
I20260619 14:45:27.193176 14665 tablet_bootstrap.cc:654] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc: Neither blocks nor log segments found. Creating new log.
I20260619 14:45:27.193387 14666 tablet_bootstrap.cc:654] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e: Neither blocks nor log segments found. Creating new log.
I20260619 14:45:27.194573 14667 tablet_bootstrap.cc:654] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff: Neither blocks nor log segments found. Creating new log.
I20260619 14:45:27.194618 14666 log.cc:824] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:27.194607 14665 log.cc:824] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:27.195813 14667 log.cc:824] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:27.196802 14665 tablet_bootstrap.cc:492] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc: No bootstrap required, opened a new log
I20260619 14:45:27.196907 14666 tablet_bootstrap.cc:492] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e: No bootstrap required, opened a new log
I20260619 14:45:27.197062 14665 ts_tablet_manager.cc:1397] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 14:45:27.197090 14666 ts_tablet_manager.cc:1397] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:45:27.197656 14667 tablet_bootstrap.cc:492] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff: No bootstrap required, opened a new log
I20260619 14:45:27.197825 14667 ts_tablet_manager.cc:1397] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:45:27.200169 14665 raft_consensus.cc:348] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } } peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260619 14:45:27.200431 14665 raft_consensus.cc:374] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:45:27.200485 14665 raft_consensus.cc:729] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1556caff2fbf42e698ccc4f464748fdc, State: Initialized, Role: FOLLOWER
I20260619 14:45:27.200964 14665 consensus_queue.cc:260] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } } peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260619 14:45:27.200856 14666 raft_consensus.cc:348] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } } peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260619 14:45:27.201144 14666 raft_consensus.cc:374] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:45:27.201205 14666 raft_consensus.cc:729] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce60de966ab4295ae6d6d43f9db4b5e, State: Initialized, Role: FOLLOWER
I20260619 14:45:27.201597 14666 consensus_queue.cc:260] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } } peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260619 14:45:27.202076 14665 ts_tablet_manager.cc:1428] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:45:27.202407 14666 ts_tablet_manager.cc:1428] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:45:27.202459 14514 heartbeater.cc:503] Master 127.0.98.62:44835 was elected leader, sending a full tablet report...
I20260619 14:45:27.202265 14667 raft_consensus.cc:348] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } } peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260619 14:45:27.202539 14667 raft_consensus.cc:374] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:45:27.202725 14648 heartbeater.cc:503] Master 127.0.98.62:44835 was elected leader, sending a full tablet report...
I20260619 14:45:27.203487 14667 raft_consensus.cc:729] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d205adbdb84e52a264ef7ce665a6ff, State: Initialized, Role: FOLLOWER
I20260619 14:45:27.203969 14667 consensus_queue.cc:260] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } } peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260619 14:45:27.204617 14380 heartbeater.cc:503] Master 127.0.98.62:44835 was elected leader, sending a full tablet report...
I20260619 14:45:27.204977 14667 ts_tablet_manager.cc:1428] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260619 14:45:27.251549 14673 raft_consensus.cc:482] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:45:27.251786 14673 raft_consensus.cc:504] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } } peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260619 14:45:27.253137 14673 leader_election.cc:288] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1556caff2fbf42e698ccc4f464748fdc (127.0.98.2:33265), 4ce60de966ab4295ae6d6d43f9db4b5e (127.0.98.3:43309)
I20260619 14:45:27.259543 14661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.253694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49229 (local address 127.0.98.2:33265)
0619 14:45:27.253855 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:27.253860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:27.254018 (+ 158us) server_negotiation.cc:407] Connection header received
0619 14:45:27.254213 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:27.254217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:27.254276 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:27.254363 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:27.254882 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.255810 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.257956 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.258614 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.258748 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.258821 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:27.259298 (+ 477us) server_negotiation.cc:299] Negotiation successful
0619 14:45:27.259381 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:45:27.259543 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.253910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54191 (local address 127.0.98.3:43309)
0619 14:45:27.254042 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:27.254049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:27.254178 (+ 129us) server_negotiation.cc:407] Connection header received
0619 14:45:27.254289 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:27.254293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:27.254352 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:27.254421 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:27.254916 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.255749 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.257956 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.258680 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.258775 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.258828 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:27.259298 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 14:45:27.259381 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:45:27.259806 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.253522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43309 (local address 127.0.98.1:54191)
0619 14:45:27.254088 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:27.254103 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:27.254200 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:27.254441 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:27.254444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:27.254456 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:27.254685 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:27.254693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.255909 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.255913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:27.257652 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:27.257666 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.258811 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.258815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:27.259096 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.259154 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 14:45:27.259248 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 14:45:27.259375 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":308,"spinlock_wait_cycles":8960,"thread_start_us":134,"threads_started":1}
I20260619 14:45:27.259814 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.253522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33265 (local address 127.0.98.1:49229)
0619 14:45:27.253895 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:27.253913 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:27.254061 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:27.254417 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:27.254421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:27.254449 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:27.254685 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:27.254693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.255919 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.255922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:27.257702 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:27.257714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.258811 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.258815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:27.259092 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.259162 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 14:45:27.259248 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 14:45:27.259374 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":264,"spinlock_wait_cycles":2944,"thread_start_us":109,"threads_started":1}
I20260619 14:45:27.260735 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb63da61480b4b9794465db0baa84d32" candidate_uuid: "66d205adbdb84e52a264ef7ce665a6ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1556caff2fbf42e698ccc4f464748fdc" is_pre_election: true
I20260619 14:45:27.260736 14602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb63da61480b4b9794465db0baa84d32" candidate_uuid: "66d205adbdb84e52a264ef7ce665a6ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" is_pre_election: true
I20260619 14:45:27.261094 14602 raft_consensus.cc:2454] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66d205adbdb84e52a264ef7ce665a6ff in term 0.
I20260619 14:45:27.261094 14468 raft_consensus.cc:2454] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66d205adbdb84e52a264ef7ce665a6ff in term 0.
I20260619 14:45:27.261619 14267 leader_election.cc:302] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1556caff2fbf42e698ccc4f464748fdc, 4ce60de966ab4295ae6d6d43f9db4b5e, 66d205adbdb84e52a264ef7ce665a6ff; no voters:
I20260619 14:45:27.261967 14673 raft_consensus.cc:2781] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:45:27.262049 14673 raft_consensus.cc:482] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:45:27.262110 14673 raft_consensus.cc:3037] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:45:27.263237 14673 raft_consensus.cc:504] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } } peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260619 14:45:27.263690 14673 leader_election.cc:288] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [CANDIDATE]: Term 1 election: Requested vote from peers 1556caff2fbf42e698ccc4f464748fdc (127.0.98.2:33265), 4ce60de966ab4295ae6d6d43f9db4b5e (127.0.98.3:43309)
I20260619 14:45:27.264103 14602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb63da61480b4b9794465db0baa84d32" candidate_uuid: "66d205adbdb84e52a264ef7ce665a6ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e"
I20260619 14:45:27.264109 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb63da61480b4b9794465db0baa84d32" candidate_uuid: "66d205adbdb84e52a264ef7ce665a6ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1556caff2fbf42e698ccc4f464748fdc"
I20260619 14:45:27.264346 14602 raft_consensus.cc:3037] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:45:27.264359 14468 raft_consensus.cc:3037] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:45:27.265419 14602 raft_consensus.cc:2454] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66d205adbdb84e52a264ef7ce665a6ff in term 1.
I20260619 14:45:27.265457 14468 raft_consensus.cc:2454] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66d205adbdb84e52a264ef7ce665a6ff in term 1.
I20260619 14:45:27.265780 14268 leader_election.cc:302] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ce60de966ab4295ae6d6d43f9db4b5e, 66d205adbdb84e52a264ef7ce665a6ff; no voters:
I20260619 14:45:27.266072 14673 raft_consensus.cc:2781] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:45:27.266346 14673 raft_consensus.cc:686] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [term 1 LEADER]: Becoming Leader. State: Replica: 66d205adbdb84e52a264ef7ce665a6ff, State: Running, Role: LEADER
I20260619 14:45:27.266724 14673 consensus_queue.cc:237] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } } peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260619 14:45:27.270179 14189 catalog_manager.cc:5495] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 66d205adbdb84e52a264ef7ce665a6ff (127.0.98.1). New cstate: current_term: 1 leader_uuid: "66d205adbdb84e52a264ef7ce665a6ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } health_report { overall_health: UNKNOWN } } }
I20260619 14:45:27.301357 14663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.297439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.0.98.1:38823)
0619 14:45:27.297719 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:27.297723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:27.297737 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:45:27.297814 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:27.297817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:27.297875 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:27.297971 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:45:27.298215 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.298769 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.299639 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.299855 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.299943 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.300281 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:45:27.300743 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:45:27.300850 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:27.301131 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 14:45:27.301207 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260619 14:45:27.307752 14468 raft_consensus.cc:1261] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc [term 1 FOLLOWER]: Refusing update from remote peer 66d205adbdb84e52a264ef7ce665a6ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:45:27.307752 14602 raft_consensus.cc:1261] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 1 FOLLOWER]: Refusing update from remote peer 66d205adbdb84e52a264ef7ce665a6ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:45:27.308562 14673 consensus_queue.cc:1035] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:45:27.308738 14676 consensus_queue.cc:1035] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:45:27.316895 14680 mvcc.cc:205] Tried to move back new op lower bound from 7298581820644392960 to 7298581820487720960. Current Snapshot: MvccSnapshot[applied={T|T < 7298581820644392960}]
I20260619 14:45:27.326526 14189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:45:27.326781 14189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:45:27.329384 14189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4: Sending DeleteTablet for 3 replicas of tablet cb63da61480b4b9794465db0baa84d32
I20260619 14:45:27.330269 14448 tablet_service.cc:1514] Processing DeleteTablet for tablet cb63da61480b4b9794465db0baa84d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:45:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57882
I20260619 14:45:27.330476 14314 tablet_service.cc:1514] Processing DeleteTablet for tablet cb63da61480b4b9794465db0baa84d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:45:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41220
I20260619 14:45:27.330471 14582 tablet_service.cc:1514] Processing DeleteTablet for tablet cb63da61480b4b9794465db0baa84d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:45:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57792
I20260619 14:45:27.330780 14689 tablet_replica.cc:330] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc: stopping tablet replica
I20260619 14:45:27.331121 14689 raft_consensus.cc:2229] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:45:27.331254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 14251
I20260619 14:45:27.331274 14690 tablet_replica.cc:330] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e: stopping tablet replica
I20260619 14:45:27.332209 14689 raft_consensus.cc:2258] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:45:27.333814 14689 ts_tablet_manager.cc:1905] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:45:27.335734 14690 raft_consensus.cc:2229] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:45:27.336092 14690 raft_consensus.cc:2258] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:45:27.337484 14689 ts_tablet_manager.cc:1918] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:45:27.337520 14690 ts_tablet_manager.cc:1905] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:45:27.337591 14689 log.cc:1196] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/cb63da61480b4b9794465db0baa84d32
I20260619 14:45:27.337946 14689 ts_tablet_manager.cc:1939] T cb63da61480b4b9794465db0baa84d32 P 1556caff2fbf42e698ccc4f464748fdc: Deleting consensus metadata
I20260619 14:45:27.339294 14173 catalog_manager.cc:4841] TS 1556caff2fbf42e698ccc4f464748fdc (127.0.98.2:33265): tablet cb63da61480b4b9794465db0baa84d32 (table test-table [id=a1b4318712ee4bd08a05ee031074b0de]) successfully deleted
W20260619 14:45:27.339488 14175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38823 (error 108)
I20260619 14:45:27.339650 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 14385
I20260619 14:45:27.340258 14690 ts_tablet_manager.cc:1918] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:45:27.340368 14690 log.cc:1196] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/cb63da61480b4b9794465db0baa84d32
I20260619 14:45:27.340744 14690 ts_tablet_manager.cc:1939] T cb63da61480b4b9794465db0baa84d32 P 4ce60de966ab4295ae6d6d43f9db4b5e: Deleting consensus metadata
I20260619 14:45:27.340932 14660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.340569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38823 (local address 127.0.0.1:41240)
0619 14:45:27.340733 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:27.340827 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 14:45:27.341161 14175 catalog_manager.cc:4584] TS 66d205adbdb84e52a264ef7ce665a6ff (127.0.98.1:38823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb63da61480b4b9794465db0baa84d32: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38823: connect: Connection refused (error 111)
I20260619 14:45:27.341974 14174 catalog_manager.cc:4841] TS 4ce60de966ab4295ae6d6d43f9db4b5e (127.0.98.3:43309): tablet cb63da61480b4b9794465db0baa84d32 (table test-table [id=a1b4318712ee4bd08a05ee031074b0de]) successfully deleted
I20260619 14:45:27.346940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 14519
I20260619 14:45:27.352890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 14157
I20260619 14:45:27.359166 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44835
--webserver_interface=127.0.98.62
--webserver_port=34857
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:45:27.471244 14693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:27.471599 14693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:27.471648 14693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:27.475785 14693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:45:27.475853 14693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:27.475873 14693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:27.475890 14693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:45:27.475905 14693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:45:27.481436 14693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:27.482581 14693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:27.483708 14693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:27.497586 14700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:27.497575 14702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:27.497566 14699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:27.497936 14693 server_base.cc:1034] running on GCE node
I20260619 14:45:27.498584 14693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:27.499881 14693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:27.501092 14693 hybrid_clock.cc:648] HybridClock initialized: now 1781880327501048 us; error 52 us; skew 500 ppm
Jun 19 14:45:27 dist-test-slave-cr22 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:27.504132 14693 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:45:27.505193 14693 webserver.cc:460] Webserver started at http://127.0.98.62:34857/ using document root <none> and password file <none>
I20260619 14:45:27.505824 14693 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:27.505955 14693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:27.510210 14693 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:45:27.512501 14709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:27.513731 14693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:45:27.513871 14693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4"
format_stamp: "Formatted at 2026-06-19 14:45:26 on dist-test-slave-cr22"
server_key: "e4c43d0fa6669418a4004d56764b0c3c"
server_key_iv: "8cdec2feda2c2b5e376ae19ca98c0583"
server_key_version: "encryptionkey@0"
I20260619 14:45:27.514469 14693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:27.538270 14693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:27.542196 14693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:27.542511 14693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:27.551024 14693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44835
I20260619 14:45:27.551355 14761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44835 every 8 connection(s)
I20260619 14:45:27.552268 14693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:45:27.556061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 14693
I20260619 14:45:27.556576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38823
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44835
--webserver_port=39045
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:45:27.557351 14762 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:45:27.559617 14762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4: Bootstrap starting.
I20260619 14:45:27.577334 14762 log.cc:824] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:27.585801 14762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:45:27.586228 14762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4: Bootstrap complete.
I20260619 14:45:27.590340 14762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44835 } }
I20260619 14:45:27.590786 14762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b8c0ed4ae4a40be8b220e575bc0f4f4, State: Initialized, Role: FOLLOWER
I20260619 14:45:27.591281 14762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44835 } }
I20260619 14:45:27.591425 14762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:45:27.591573 14762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:45:27.591691 14762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:45:27.593588 14762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44835 } }
I20260619 14:45:27.593938 14762 leader_election.cc:302] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b8c0ed4ae4a40be8b220e575bc0f4f4; no voters:
I20260619 14:45:27.594280 14762 leader_election.cc:288] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:45:27.594394 14766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:45:27.594605 14766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 2 LEADER]: Becoming Leader. State: Replica: 9b8c0ed4ae4a40be8b220e575bc0f4f4, State: Running, Role: LEADER
I20260619 14:45:27.594889 14766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44835 } }
I20260619 14:45:27.595537 14762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:45:27.595875 14766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b8c0ed4ae4a40be8b220e575bc0f4f4. Latest consensus state: current_term: 2 leader_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44835 } } }
I20260619 14:45:27.595989 14766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [sys.catalog]: This master's current role is: LEADER
I20260619 14:45:27.596400 14767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44835 } } }
I20260619 14:45:27.596733 14767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [sys.catalog]: This master's current role is: LEADER
I20260619 14:45:27.597774 14770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:45:27.601707 14770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:45:27.602219 14770 catalog_manager.cc:1259] Loaded cluster ID: f03ea41ece344956a7a6e1593373d64b
I20260619 14:45:27.602293 14770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:45:27.603364 14770 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 14:45:27.606133 14770 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 14:45:27.610703 14770 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:45:27.611533 14770 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4: Loaded TSK: 0
I20260619 14:45:27.612497 14770 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:45:27.688835 14764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:27.689117 14764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:27.689164 14764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:27.693910 14764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:27.693974 14764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:27.694084 14764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:45:27.700528 14764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39045
--tserver_master_addrs=127.0.98.62:44835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:27.701624 14764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:27.702684 14764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:27.714437 14789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:27.714437 14790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:27.714438 14792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:27.715375 14764 server_base.cc:1034] running on GCE node
I20260619 14:45:27.715896 14764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:27.716714 14764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:27.717893 14764 hybrid_clock.cc:648] HybridClock initialized: now 1781880327717864 us; error 55 us; skew 500 ppm
Jun 19 14:45:27 dist-test-slave-cr22 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:27.721021 14764 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:45:27.722070 14764 webserver.cc:460] Webserver started at http://127.0.98.1:39045/ using document root <none> and password file <none>
I20260619 14:45:27.722685 14764 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:27.722764 14764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:27.728201 14764 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:45:27.730672 14799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:27.732151 14764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:45:27.732290 14764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "66d205adbdb84e52a264ef7ce665a6ff"
format_stamp: "Formatted at 2026-06-19 14:45:26 on dist-test-slave-cr22"
server_key: "d823822b4f050fc225970969263199e0"
server_key_iv: "9fc7244a758ba683f5d955360dd51d7c"
server_key_version: "encryptionkey@0"
I20260619 14:45:27.732875 14764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:27.754498 14764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:27.758076 14764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:27.758311 14764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:27.758988 14764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:45:27.760442 14806 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:45:27.765033 14764 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:45:27.765112 14764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260619 14:45:27.765165 14764 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:45:27.766995 14764 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:45:27.767059 14764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 14:45:27.767318 14806 tablet_bootstrap.cc:492] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff: Bootstrap starting.
I20260619 14:45:27.779243 14764 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38823
I20260619 14:45:27.779426 14913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38823 every 8 connection(s)
I20260619 14:45:27.780663 14764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:45:27.784583 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 14764
I20260619 14:45:27.785163 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33265
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44835
--webserver_port=35891
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:45:27.791332 14806 log.cc:824] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:45:27 dist-test-slave-cr22 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:45:27.800316 14806 tablet_bootstrap.cc:492] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:45:27.800824 14806 tablet_bootstrap.cc:492] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff: Bootstrap complete.
I20260619 14:45:27.802188 14917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.783000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:39483)
0619 14:45:27.783634 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:27.783679 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:27.784610 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:27.786006 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:27.786015 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:27.786489 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:27.787193 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:27.787219 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.789424 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.789427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:27.790419 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:27.790429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.791615 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.791623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:27.791883 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.792659 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:27.792689 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:27.794919 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:27.798138 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:27.798144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:27.798157 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:27.798536 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:27.799025 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:27.799028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:27.799030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:27.799203 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:27.800115 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:27.800121 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:27.800466 (+ 345us) client_negotiation.cc:769] Sending connection context
0619 14:45:27.800704 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 14:45:27.801071 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":158,"threads_started":1}
I20260619 14:45:27.802392 14806 ts_tablet_manager.cc:1397] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff: Time spent bootstrapping tablet: real 0.035s user 0.023s sys 0.012s
I20260619 14:45:27.803004 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.783014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39483 (local address 127.0.98.62:44835)
0619 14:45:27.783419 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:27.783466 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:27.783813 (+ 347us) server_negotiation.cc:407] Connection header received
0619 14:45:27.785119 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:27.785130 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:27.785570 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:27.785951 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:27.787470 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.789238 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.790620 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.791364 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.791617 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.795166 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:27.795176 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:27.795199 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:27.795246 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:27.797979 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:27.798682 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:27.798693 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:27.798700 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:27.798887 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:27.799322 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:27.799327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:27.799331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:27.799955 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:27.800112 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:27.800941 (+ 829us) server_negotiation.cc:299] Negotiation successful
0619 14:45:27.801715 (+ 774us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":142,"threads_started":1}
I20260619 14:45:27.805691 14914 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44835
I20260619 14:45:27.805965 14914 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:27.805640 14806 raft_consensus.cc:348] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } } peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260619 14:45:27.806260 14806 raft_consensus.cc:729] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d205adbdb84e52a264ef7ce665a6ff, State: Initialized, Role: FOLLOWER
I20260619 14:45:27.806558 14914 heartbeater.cc:511] Master 127.0.98.62:44835 requested a full tablet report, sending...
I20260619 14:45:27.806815 14806 consensus_queue.cc:260] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } } peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260619 14:45:27.807862 14806 ts_tablet_manager.cc:1428] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:45:27.809523 14726 ts_manager.cc:194] Registered new tserver with Master: 66d205adbdb84e52a264ef7ce665a6ff (127.0.98.1:38823)
I20260619 14:45:27.813079 14726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39483
I20260619 14:45:27.815973 14914 heartbeater.cc:503] Master 127.0.98.62:44835 was elected leader, sending a full tablet report...
Jun 19 14:45:27 dist-test-slave-cr22 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:45:27.823978 14922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.812756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.0.98.1:38823)
0619 14:45:27.813258 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:27.813271 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:27.813636 (+ 365us) server_negotiation.cc:407] Connection header received
0619 14:45:27.813853 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:27.813859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:27.814059 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:27.814208 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:27.814805 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.815610 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.816608 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.817086 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.817195 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.820051 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:27.820059 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:27.820070 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:27.820097 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:27.821804 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:27.822397 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:27.822402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:27.822403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:27.822483 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:27.822841 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:27.822847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:27.822852 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:27.823113 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:27.823217 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:27.823693 (+ 476us) server_negotiation.cc:299] Negotiation successful
0619 14:45:27.823842 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":102,"threads_started":1}
I20260619 14:45:27.824174 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.812560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38823 (local address 127.0.0.1:41254)
0619 14:45:27.813465 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:27.813520 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:27.813686 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:27.814207 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:27.814213 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:27.814582 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:27.814680 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:27.814687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.815726 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.815730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:27.816483 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:27.816495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.817203 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.817221 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:27.817344 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.817985 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:27.818001 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:27.819907 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:27.821925 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:27.821936 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:27.821942 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:27.822264 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:27.822596 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:27.822599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:27.822602 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:27.822704 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:27.823232 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:27.823239 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:27.823379 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 14:45:27.823758 (+ 379us) client_negotiation.cc:240] Negotiation successful
0619 14:45:27.824001 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":101,"threads_started":1}
I20260619 14:45:27.825351 14848 tablet_service.cc:1514] Processing DeleteTablet for tablet cb63da61480b4b9794465db0baa84d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:45:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41254
I20260619 14:45:27.826382 14923 tablet_replica.cc:330] stopping tablet replica
I20260619 14:45:27.826687 14923 raft_consensus.cc:2229] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:45:27.826939 14923 raft_consensus.cc:2258] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:45:27.827968 14923 ts_tablet_manager.cc:1905] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:45:27.831497 14923 ts_tablet_manager.cc:1918] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:45:27.831598 14923 log.cc:1196] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/wal/wals/cb63da61480b4b9794465db0baa84d32
I20260619 14:45:27.831876 14923 ts_tablet_manager.cc:1939] T cb63da61480b4b9794465db0baa84d32 P 66d205adbdb84e52a264ef7ce665a6ff: Deleting consensus metadata
I20260619 14:45:27.833771 14712 catalog_manager.cc:4841] TS 66d205adbdb84e52a264ef7ce665a6ff (127.0.98.1:38823): tablet cb63da61480b4b9794465db0baa84d32 (table test-table [id=a1b4318712ee4bd08a05ee031074b0de]) successfully deleted
W20260619 14:45:27.914172 14918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:27.914495 14918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:27.914572 14918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:27.918886 14918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:27.918979 14918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:27.919100 14918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:45:27.924966 14918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35891
--tserver_master_addrs=127.0.98.62:44835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:27.926007 14918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:27.927017 14918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:27.936880 14929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:27.936905 14928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:27.936998 14918 server_base.cc:1034] running on GCE node
W20260619 14:45:27.936880 14931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:27.937594 14918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:27.938262 14918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:27.939523 14918 hybrid_clock.cc:648] HybridClock initialized: now 1781880327939494 us; error 48 us; skew 500 ppm
Jun 19 14:45:27 dist-test-slave-cr22 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:27.942346 14918 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:45:27.943259 14918 webserver.cc:460] Webserver started at http://127.0.98.2:35891/ using document root <none> and password file <none>
I20260619 14:45:27.943938 14918 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:27.944052 14918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:27.947973 14918 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:45:27.950137 14938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:27.951650 14918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:45:27.951781 14918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "1556caff2fbf42e698ccc4f464748fdc"
format_stamp: "Formatted at 2026-06-19 14:45:26 on dist-test-slave-cr22"
server_key: "bf438cc2aac8fea2d1f056f098a08f0f"
server_key_iv: "bc09a8c7d605a910eba123457dd387ca"
server_key_version: "encryptionkey@0"
I20260619 14:45:27.952329 14918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:27.971670 14918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:27.974563 14918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:27.974777 14918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:27.975389 14918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:45:27.976229 14918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:45:27.976300 14918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:27.976382 14918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:45:27.976433 14918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:27.985426 14918 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33265
I20260619 14:45:27.985450 15051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33265 every 8 connection(s)
I20260619 14:45:27.986394 14918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 14:45:27 dist-test-slave-cr22 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:45:27.996147 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 14918
I20260619 14:45:27.996711 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43309
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44835
--webserver_port=40111
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:45:28.001148 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.988326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33393 (local address 127.0.98.62:44835)
0619 14:45:27.988501 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:27.988513 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:27.989330 (+ 817us) server_negotiation.cc:407] Connection header received
0619 14:45:27.990636 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:27.990639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:27.990687 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:27.990761 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:27.992032 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.992622 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.993772 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.993995 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.994078 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.997013 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:27.997023 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:27.997025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:27.997054 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:27.998884 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:27.999389 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:27.999394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:27.999396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:27.999511 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:27.999922 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:27.999925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:27.999926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:28.000134 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:28.000211 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:28.000814 (+ 603us) server_negotiation.cc:299] Negotiation successful
0619 14:45:28.000966 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:45:28.002308 15054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:27.988734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:33393)
0619 14:45:27.989183 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:27.989223 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:27.990412 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:27.990920 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:27.990926 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:27.991286 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:27.991886 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:27.991898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.992745 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.992748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:27.993653 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:27.993662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:27.994133 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:27.994136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:27.994290 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:27.994888 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:27.994903 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:27.996798 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:27.999003 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:27.999008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:27.999019 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:27.999289 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:27.999637 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:27.999643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:27.999648 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:27.999827 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:28.000240 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:28.000249 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:28.000558 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 14:45:28.000775 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 14:45:28.001144 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":110,"threads_started":1}
I20260619 14:45:28.003573 15052 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44835
I20260619 14:45:28.003839 15052 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:28.004459 15052 heartbeater.cc:511] Master 127.0.98.62:44835 requested a full tablet report, sending...
I20260619 14:45:28.005602 14726 ts_manager.cc:194] Registered new tserver with Master: 1556caff2fbf42e698ccc4f464748fdc (127.0.98.2:33265)
I20260619 14:45:28.006172 14726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33393
W20260619 14:45:28.116665 15055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:28.116971 15055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:28.117067 15055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:28.121318 15055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:28.121506 15055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:28.121621 15055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:45:28.127646 15055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40111
--tserver_master_addrs=127.0.98.62:44835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:28.128754 15055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:28.129784 15055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:28.140278 15060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:28.140304 15063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:28.140342 15061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:28.140592 15055 server_base.cc:1034] running on GCE node
I20260619 14:45:28.141006 15055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:28.141603 15055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:28.142771 15055 hybrid_clock.cc:648] HybridClock initialized: now 1781880328142751 us; error 31 us; skew 500 ppm
Jun 19 14:45:28 dist-test-slave-cr22 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:28.145581 15055 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:45:28.146463 15055 webserver.cc:460] Webserver started at http://127.0.98.3:40111/ using document root <none> and password file <none>
I20260619 14:45:28.147023 15055 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:28.147153 15055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:28.151330 15055 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:45:28.153443 15070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:28.154942 15055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:45:28.155134 15055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "4ce60de966ab4295ae6d6d43f9db4b5e"
format_stamp: "Formatted at 2026-06-19 14:45:27 on dist-test-slave-cr22"
server_key: "7d935537a5f474b8c18bc4d31abc6176"
server_key_iv: "60efcef9ff761cb5665033f034da0184"
server_key_version: "encryptionkey@0"
I20260619 14:45:28.155798 15055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:28.174902 15055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:28.178351 15055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:28.178560 15055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:28.179164 15055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:45:28.180053 15055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:45:28.180124 15055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:28.180186 15055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:45:28.180233 15055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:28.189843 15055 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43309
I20260619 14:45:28.189875 15183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43309 every 8 connection(s)
I20260619 14:45:28.190845 15055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:45:28.194561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 15055
Jun 19 14:45:28 dist-test-slave-cr22 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:45:28.203684 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:28.195349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60216 (local address 127.0.98.62:44835)
0619 14:45:28.195656 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:28.195661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:28.195792 (+ 131us) server_negotiation.cc:407] Connection header received
0619 14:45:28.196018 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:28.196025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:28.196079 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:28.196172 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:28.197065 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.197664 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.198552 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.198802 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.198912 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:28.200417 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:28.200420 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:28.200425 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:28.200453 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:28.202050 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:28.202458 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:28.202463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:28.202465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:28.202547 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:28.202823 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:28.202826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:28.202828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:28.203000 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:28.203223 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:28.203355 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 14:45:28.203505 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":104,"threads_started":1}
I20260619 14:45:28.205688 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:28.192808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32787 (local address 127.0.98.62:44835)
0619 14:45:28.192961 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:28.192965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:28.193778 (+ 813us) server_negotiation.cc:407] Connection header received
0619 14:45:28.195192 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:28.195195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:28.195253 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:28.195354 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:28.196817 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.197436 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.198646 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.198870 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.198982 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:28.201756 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:28.201760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:28.201762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:28.201793 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:28.203314 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:28.204013 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:28.204015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:28.204017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:28.204090 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:28.204508 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:28.204520 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:28.204522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:28.204717 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:28.204797 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:28.205444 (+ 647us) server_negotiation.cc:299] Negotiation successful
0619 14:45:28.205549 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:45:28.206499 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:28.193179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:32787)
0619 14:45:28.193628 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:28.193670 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:28.194955 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:28.195613 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:28.195622 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:28.196059 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:28.196674 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:28.196690 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.197559 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.197563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:28.198508 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:28.198519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.199023 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.199027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:28.199237 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:28.199832 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:28.199848 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:28.201565 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:28.203477 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:28.203485 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:28.203503 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:28.203899 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:28.204181 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:28.204184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:28.204186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:28.204369 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:28.204828 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:28.204835 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:28.205175 (+ 340us) client_negotiation.cc:769] Sending connection context
0619 14:45:28.205349 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 14:45:28.205619 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":116,"threads_started":1}
I20260619 14:45:28.207721 15184 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44835
I20260619 14:45:28.207937 15184 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:28.208367 15184 heartbeater.cc:511] Master 127.0.98.62:44835 requested a full tablet report, sending...
I20260619 14:45:28.209316 14726 ts_manager.cc:194] Registered new tserver with Master: 4ce60de966ab4295ae6d6d43f9db4b5e (127.0.98.3:43309)
I20260619 14:45:28.209880 14726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32787
I20260619 14:45:28.217813 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:45:28.222329 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:28.218469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60222 (local address 127.0.98.62:44835)
0619 14:45:28.218604 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:28.218609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:28.218845 (+ 236us) server_negotiation.cc:407] Connection header received
0619 14:45:28.219010 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:28.219013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:28.219178 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:28.219265 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:45:28.219517 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.220036 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.220951 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.221178 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.221258 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:28.221514 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:45:28.221912 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:45:28.222004 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:28.222112 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 14:45:28.222157 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:45:28.222972 14726 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:45:28.226277 14726 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:45:28.233973 14848 tablet_service.cc:1467] Processing CreateTablet for tablet e6629348a364478aa4c155ae371a4808 (DEFAULT_TABLE table=test-table [id=4d12ddec532944c39a32f6c49c18d4bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:45:28.234664 14848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6629348a364478aa4c155ae371a4808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:28.237758 14806 tablet_bootstrap.cc:492] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff: Bootstrap starting.
I20260619 14:45:28.238580 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:28.232904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43309 (local address 127.0.0.1:57804)
0619 14:45:28.233066 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:28.233081 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:28.233158 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:28.234058 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:28.234061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:28.234087 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:28.234194 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:28.234203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.235507 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.235510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:28.236747 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:28.236757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.238113 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.238117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:28.238210 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:28.238249 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 14:45:28.238336 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:45:28.238424 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 14:45:28.238723 14806 tablet_bootstrap.cc:654] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff: Neither blocks nor log segments found. Creating new log.
I20260619 14:45:28.238603 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:28.233149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57804 (local address 127.0.98.3:43309)
0619 14:45:28.233495 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:28.233515 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:28.233541 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:45:28.233631 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:28.233639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:28.233879 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:28.234078 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:28.234322 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.235295 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.236877 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.237958 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.238112 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:28.238262 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:28.238372 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 14:45:28.238475 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":185,"threads_started":1}
I20260619 14:45:28.239213 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:28.233355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33265 (local address 127.0.0.1:57890)
0619 14:45:28.233716 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:28.233730 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:28.233817 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:28.234703 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:28.234706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:28.234725 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:28.234811 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:28.234816 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.236187 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.236191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:28.237155 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:28.237188 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.238872 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.238876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:28.238971 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:28.238995 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:45:28.239068 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 14:45:28.239121 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":83,"threads_started":1}
I20260619 14:45:28.239401 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:28.233750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57890 (local address 127.0.98.2:33265)
0619 14:45:28.234101 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:28.234113 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:28.234137 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:45:28.234356 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:28.234361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:28.234548 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:28.234699 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:28.234945 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.236051 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.237314 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.238766 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.238872 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:28.239017 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:28.239142 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 14:45:28.239240 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":108,"threads_started":1}
I20260619 14:45:28.240331 14806 tablet_bootstrap.cc:492] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff: No bootstrap required, opened a new log
I20260619 14:45:28.240456 14806 ts_tablet_manager.cc:1397] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:45:28.240886 14806 raft_consensus.cc:348] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260619 14:45:28.241012 14806 raft_consensus.cc:374] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:45:28.241058 14806 raft_consensus.cc:729] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d205adbdb84e52a264ef7ce665a6ff, State: Initialized, Role: FOLLOWER
I20260619 14:45:28.241209 14806 consensus_queue.cc:260] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260619 14:45:28.242170 14806 ts_tablet_manager.cc:1428] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:45:28.242479 14986 tablet_service.cc:1467] Processing CreateTablet for tablet e6629348a364478aa4c155ae371a4808 (DEFAULT_TABLE table=test-table [id=4d12ddec532944c39a32f6c49c18d4bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:45:28.242479 15118 tablet_service.cc:1467] Processing CreateTablet for tablet e6629348a364478aa4c155ae371a4808 (DEFAULT_TABLE table=test-table [id=4d12ddec532944c39a32f6c49c18d4bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:45:28.243629 14986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6629348a364478aa4c155ae371a4808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:28.243639 15118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6629348a364478aa4c155ae371a4808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:28.252560 15197 tablet_bootstrap.cc:492] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: Bootstrap starting.
I20260619 14:45:28.255203 15197 tablet_bootstrap.cc:654] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: Neither blocks nor log segments found. Creating new log.
I20260619 14:45:28.255524 15198 tablet_bootstrap.cc:492] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc: Bootstrap starting.
I20260619 14:45:28.256616 15197 log.cc:824] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:28.258667 15198 tablet_bootstrap.cc:654] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc: Neither blocks nor log segments found. Creating new log.
I20260619 14:45:28.259544 15197 tablet_bootstrap.cc:492] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: No bootstrap required, opened a new log
I20260619 14:45:28.259653 15198 log.cc:824] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:28.259754 15197 ts_tablet_manager.cc:1397] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260619 14:45:28.261406 15198 tablet_bootstrap.cc:492] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc: No bootstrap required, opened a new log
I20260619 14:45:28.261610 15198 ts_tablet_manager.cc:1397] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20260619 14:45:28.262931 15197 raft_consensus.cc:348] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260619 14:45:28.263242 15197 raft_consensus.cc:374] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:45:28.263307 15197 raft_consensus.cc:729] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce60de966ab4295ae6d6d43f9db4b5e, State: Initialized, Role: FOLLOWER
I20260619 14:45:28.263739 15197 consensus_queue.cc:260] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260619 14:45:28.264372 15184 heartbeater.cc:503] Master 127.0.98.62:44835 was elected leader, sending a full tablet report...
I20260619 14:45:28.264647 15197 ts_tablet_manager.cc:1428] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:45:28.265125 15198 raft_consensus.cc:348] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260619 14:45:28.265377 15198 raft_consensus.cc:374] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:45:28.265431 15198 raft_consensus.cc:729] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1556caff2fbf42e698ccc4f464748fdc, State: Initialized, Role: FOLLOWER
I20260619 14:45:28.265858 15198 consensus_queue.cc:260] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260619 14:45:28.266558 15052 heartbeater.cc:503] Master 127.0.98.62:44835 was elected leader, sending a full tablet report...
I20260619 14:45:28.266816 15198 ts_tablet_manager.cc:1428] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260619 14:45:28.281966 14915 tablet.cc:2369] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:45:28.441869 15185 tablet.cc:2369] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:45:28.487675 15053 tablet.cc:2369] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:45:28.630779 14920 raft_consensus.cc:482] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:45:28.631091 14920 raft_consensus.cc:504] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260619 14:45:28.632364 14920 leader_election.cc:288] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ce60de966ab4295ae6d6d43f9db4b5e (127.0.98.3:43309), 1556caff2fbf42e698ccc4f464748fdc (127.0.98.2:33265)
I20260619 14:45:28.637998 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:28.632732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39497 (local address 127.0.98.3:43309)
0619 14:45:28.632919 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:28.632924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:28.633141 (+ 217us) server_negotiation.cc:407] Connection header received
0619 14:45:28.633292 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:28.633295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:28.633345 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:28.633435 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:28.633849 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.634725 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.636437 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.637193 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.637287 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:28.637339 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:28.637787 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 14:45:28.637862 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 14:45:28.637998 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:28.632888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46427 (local address 127.0.98.2:33265)
0619 14:45:28.633021 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:28.633024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:28.633201 (+ 177us) server_negotiation.cc:407] Connection header received
0619 14:45:28.633278 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:28.633281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:28.633340 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:28.633437 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:28.633856 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.634646 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.636437 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.637133 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.637265 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:28.637339 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:28.637787 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 14:45:28.637865 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:45:28.638135 15204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:28.632749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33265 (local address 127.0.98.1:46427)
0619 14:45:28.633094 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:28.633106 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:28.633182 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:28.633472 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:28.633475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:28.633518 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:28.633716 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:28.633723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.634823 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.634829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:28.636277 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:28.636289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.637395 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.637399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:28.637544 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:28.637654 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 14:45:28.637741 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:45:28.637907 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":217,"spinlock_wait_cycles":6272,"thread_start_us":83,"threads_started":1}
I20260619 14:45:28.638157 15203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:28.632566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43309 (local address 127.0.98.1:39497)
0619 14:45:28.633011 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:28.633027 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:28.633166 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:28.633472 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:28.633475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:28.633517 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:28.633716 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:28.633723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.634840 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.634842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:28.636277 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:28.636289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.637395 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.637399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:28.637593 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:28.637656 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 14:45:28.637741 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 14:45:28.637907 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":247,"spinlock_wait_cycles":28288,"thread_start_us":95,"threads_started":1}
I20260619 14:45:28.638844 15138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6629348a364478aa4c155ae371a4808" candidate_uuid: "66d205adbdb84e52a264ef7ce665a6ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" is_pre_election: true
I20260619 14:45:28.638844 15006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6629348a364478aa4c155ae371a4808" candidate_uuid: "66d205adbdb84e52a264ef7ce665a6ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1556caff2fbf42e698ccc4f464748fdc" is_pre_election: true
I20260619 14:45:28.639176 15138 raft_consensus.cc:2454] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66d205adbdb84e52a264ef7ce665a6ff in term 0.
I20260619 14:45:28.639176 15006 raft_consensus.cc:2454] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66d205adbdb84e52a264ef7ce665a6ff in term 0.
I20260619 14:45:28.639724 14801 leader_election.cc:302] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ce60de966ab4295ae6d6d43f9db4b5e, 66d205adbdb84e52a264ef7ce665a6ff; no voters:
I20260619 14:45:28.640092 14920 raft_consensus.cc:2781] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:45:28.640198 14920 raft_consensus.cc:482] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:45:28.640255 14920 raft_consensus.cc:3037] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:45:28.641300 14920 raft_consensus.cc:504] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260619 14:45:28.641705 14920 leader_election.cc:288] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [CANDIDATE]: Term 1 election: Requested vote from peers 4ce60de966ab4295ae6d6d43f9db4b5e (127.0.98.3:43309), 1556caff2fbf42e698ccc4f464748fdc (127.0.98.2:33265)
I20260619 14:45:28.642113 15006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6629348a364478aa4c155ae371a4808" candidate_uuid: "66d205adbdb84e52a264ef7ce665a6ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1556caff2fbf42e698ccc4f464748fdc"
I20260619 14:45:28.642196 15138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6629348a364478aa4c155ae371a4808" candidate_uuid: "66d205adbdb84e52a264ef7ce665a6ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e"
I20260619 14:45:28.642304 15006 raft_consensus.cc:3037] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:45:28.642349 15138 raft_consensus.cc:3037] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:45:28.643359 15138 raft_consensus.cc:2454] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66d205adbdb84e52a264ef7ce665a6ff in term 1.
I20260619 14:45:28.643442 15006 raft_consensus.cc:2454] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66d205adbdb84e52a264ef7ce665a6ff in term 1.
I20260619 14:45:28.643704 14801 leader_election.cc:302] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ce60de966ab4295ae6d6d43f9db4b5e, 66d205adbdb84e52a264ef7ce665a6ff; no voters:
I20260619 14:45:28.643867 14920 raft_consensus.cc:2781] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:45:28.644071 14920 raft_consensus.cc:686] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 1 LEADER]: Becoming Leader. State: Replica: 66d205adbdb84e52a264ef7ce665a6ff, State: Running, Role: LEADER
I20260619 14:45:28.644361 14920 consensus_queue.cc:237] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260619 14:45:28.647517 14725 catalog_manager.cc:5495] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 66d205adbdb84e52a264ef7ce665a6ff (127.0.98.1). New cstate: current_term: 1 leader_uuid: "66d205adbdb84e52a264ef7ce665a6ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } health_report { overall_health: HEALTHY } } }
I20260619 14:45:28.699448 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:28.695605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.0.98.1:38823)
0619 14:45:28.695955 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:28.695961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:28.695975 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:45:28.696050 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:28.696052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:28.696113 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:28.696288 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:45:28.696510 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.697179 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.698058 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:28.698279 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:28.698360 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:28.698611 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:45:28.699039 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:45:28.699125 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:28.699194 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 14:45:28.699244 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":183,"threads_started":1}
I20260619 14:45:28.703554 15138 raft_consensus.cc:1261] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 1 FOLLOWER]: Refusing update from remote peer 66d205adbdb84e52a264ef7ce665a6ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:45:28.703554 15006 raft_consensus.cc:1261] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc [term 1 FOLLOWER]: Refusing update from remote peer 66d205adbdb84e52a264ef7ce665a6ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:45:28.704306 15205 consensus_queue.cc:1035] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:45:28.704478 14920 consensus_queue.cc:1035] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:45:28.712415 15210 mvcc.cc:205] Tried to move back new op lower bound from 7298581826363133952 to 7298581826129989632. Current Snapshot: MvccSnapshot[applied={T|T < 7298581826363133952}]
I20260619 14:45:28.712409 15209 mvcc.cc:205] Tried to move back new op lower bound from 7298581826363133952 to 7298581826129989632. Current Snapshot: MvccSnapshot[applied={T|T < 7298581826363133952}]
I20260619 14:45:28.721144 14725 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:45:28.721326 14725 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:45:28.723873 14725 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4: Sending DeleteTablet for 3 replicas of tablet e6629348a364478aa4c155ae371a4808
I20260619 14:45:28.724642 14848 tablet_service.cc:1514] Processing DeleteTablet for tablet e6629348a364478aa4c155ae371a4808 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:45:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41254
I20260619 14:45:28.724654 15118 tablet_service.cc:1514] Processing DeleteTablet for tablet e6629348a364478aa4c155ae371a4808 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:45:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57804
I20260619 14:45:28.724732 14986 tablet_service.cc:1514] Processing DeleteTablet for tablet e6629348a364478aa4c155ae371a4808 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:45:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57890
I20260619 14:45:28.725296 15218 tablet_replica.cc:330] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff: stopping tablet replica
I20260619 14:45:28.725391 15219 tablet_replica.cc:330] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: stopping tablet replica
I20260619 14:45:28.725479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 14764
I20260619 14:45:28.725523 15218 raft_consensus.cc:2229] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:45:28.731755 15219 raft_consensus.cc:2229] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 14:45:28.732254 14712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38823 (error 108)
I20260619 14:45:28.732406 15219 raft_consensus.cc:2258] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:45:28.732810 15220 tablet_replica.cc:330] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc: stopping tablet replica
I20260619 14:45:28.732872 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 14918
I20260619 14:45:28.733224 15220 raft_consensus.cc:2229] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:45:28.734954 15219 ts_tablet_manager.cc:1905] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:45:28.735304 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:28.734845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38823 (local address 127.0.0.1:41274)
0619 14:45:28.735084 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:28.735184 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260619 14:45:28.735538 14712 catalog_manager.cc:4584] TS 66d205adbdb84e52a264ef7ce665a6ff (127.0.98.1:38823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6629348a364478aa4c155ae371a4808: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38823: connect: Connection refused (error 111)
I20260619 14:45:28.737859 15219 ts_tablet_manager.cc:1918] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:45:28.737974 15219 log.cc:1196] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/e6629348a364478aa4c155ae371a4808
I20260619 14:45:28.738282 15219 ts_tablet_manager.cc:1939] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: Deleting consensus metadata
I20260619 14:45:28.739570 14711 catalog_manager.cc:4841] TS 4ce60de966ab4295ae6d6d43f9db4b5e (127.0.98.3:43309): tablet e6629348a364478aa4c155ae371a4808 (table test-table [id=4d12ddec532944c39a32f6c49c18d4bd]) successfully deleted
I20260619 14:45:28.740223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 15055
I20260619 14:45:28.741176 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:28.740375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33265 (local address 127.0.0.1:57894)
0619 14:45:28.740989 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:28.741059 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508}
I20260619 14:45:28.746150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 14693
I20260619 14:45:28.753393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44835
--webserver_interface=127.0.98.62
--webserver_port=34857
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:45:28.870434 15222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:28.870842 15222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:28.870955 15222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:28.875244 15222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:45:28.875350 15222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:28.875397 15222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:28.875474 15222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:45:28.875531 15222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:45:28.881577 15222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:28.882822 15222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:28.883952 15222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:28.893587 15228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:28.893587 15230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:28.893692 15227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:28.893630 15222 server_base.cc:1034] running on GCE node
I20260619 14:45:28.894424 15222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:28.895673 15222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:28.896871 15222 hybrid_clock.cc:648] HybridClock initialized: now 1781880328896838 us; error 46 us; skew 500 ppm
Jun 19 14:45:28 dist-test-slave-cr22 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:28.900094 15222 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:45:28.901079 15222 webserver.cc:460] Webserver started at http://127.0.98.62:34857/ using document root <none> and password file <none>
I20260619 14:45:28.901656 15222 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:28.901746 15222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:28.905750 15222 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:45:28.907989 15237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:28.909318 15222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:45:28.909478 15222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4"
format_stamp: "Formatted at 2026-06-19 14:45:26 on dist-test-slave-cr22"
server_key: "e4c43d0fa6669418a4004d56764b0c3c"
server_key_iv: "8cdec2feda2c2b5e376ae19ca98c0583"
server_key_version: "encryptionkey@0"
I20260619 14:45:28.910116 15222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:28.925100 15222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:28.928547 15222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:28.928807 15222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:28.937402 15222 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44835
I20260619 14:45:28.937728 15289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44835 every 8 connection(s)
I20260619 14:45:28.938706 15222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:45:28.940464 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 15222
I20260619 14:45:28.940914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38823
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44835
--webserver_port=39045
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:45:28.944211 15290 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:45:28.946115 15290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4: Bootstrap starting.
I20260619 14:45:28.964293 15290 log.cc:824] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:28.975878 15290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:45:28.976291 15290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4: Bootstrap complete.
I20260619 14:45:28.981035 15290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44835 } }
I20260619 14:45:28.981459 15290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b8c0ed4ae4a40be8b220e575bc0f4f4, State: Initialized, Role: FOLLOWER
I20260619 14:45:28.982034 15290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44835 } }
I20260619 14:45:28.982185 15290 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:45:28.982276 15290 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:45:28.982399 15290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:45:28.984477 15290 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44835 } }
I20260619 14:45:28.984829 15290 leader_election.cc:302] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b8c0ed4ae4a40be8b220e575bc0f4f4; no voters:
I20260619 14:45:28.985211 15290 leader_election.cc:288] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 14:45:28.985312 15294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 14:45:28.985512 15294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [term 3 LEADER]: Becoming Leader. State: Replica: 9b8c0ed4ae4a40be8b220e575bc0f4f4, State: Running, Role: LEADER
I20260619 14:45:28.985806 15294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44835 } }
I20260619 14:45:28.986482 15290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:45:28.986838 15294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b8c0ed4ae4a40be8b220e575bc0f4f4. Latest consensus state: current_term: 3 leader_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44835 } } }
I20260619 14:45:28.986979 15294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [sys.catalog]: This master's current role is: LEADER
I20260619 14:45:28.987350 15298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:45:28.987358 15295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b8c0ed4ae4a40be8b220e575bc0f4f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44835 } } }
I20260619 14:45:28.987646 15295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b8c0ed4ae4a40be8b220e575bc0f4f4 [sys.catalog]: This master's current role is: LEADER
I20260619 14:45:28.992722 15298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:45:28.993446 15298 catalog_manager.cc:1259] Loaded cluster ID: f03ea41ece344956a7a6e1593373d64b
I20260619 14:45:28.993534 15298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 14:45:28.994731 15298 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6a1adbd9ad google::LogMessage::Fail() at ??:0
@ 0x7f6a1adc088a google::LogMessage::SendToLog() at ??:0
@ 0x7f6a1adbd4de google::LogMessage::Flush() at ??:0
@ 0x7f6a1adbea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6a256fcfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6a256fdd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6a256fa579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f6a2573e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6a1d00f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6a1d00fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f6a1d0113f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6a1d000512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6a1e6446db start_thread at ??:0
@ 0x7f6a198bb61f clone at ??:0
W20260619 14:45:29.096012 15292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:29.096302 15292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:29.096366 15292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:29.100817 15292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:29.100888 15292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:29.100952 15292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:45:29.106586 15292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39045
--tserver_master_addrs=127.0.98.62:44835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:29.107908 15292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:29.108978 15292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:29.120373 15334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:29.120386 15331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:29.120381 15332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:29.121667 15292 server_base.cc:1034] running on GCE node
I20260619 14:45:29.122169 15292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:29.122845 15292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:29.124060 15292 hybrid_clock.cc:648] HybridClock initialized: now 1781880329124032 us; error 43 us; skew 500 ppm
Jun 19 14:45:29 dist-test-slave-cr22 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:29.127044 15292 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:45:29.128157 15292 webserver.cc:460] Webserver started at http://127.0.98.1:39045/ using document root <none> and password file <none>
I20260619 14:45:29.128736 15292 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:29.128798 15292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:29.133011 15292 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:45:29.135583 15341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:29.137063 15292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:45:29.137209 15292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "66d205adbdb84e52a264ef7ce665a6ff"
format_stamp: "Formatted at 2026-06-19 14:45:26 on dist-test-slave-cr22"
server_key: "d823822b4f050fc225970969263199e0"
server_key_iv: "9fc7244a758ba683f5d955360dd51d7c"
server_key_version: "encryptionkey@0"
I20260619 14:45:29.137995 15292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:29.157631 15292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:29.161012 15292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:29.161223 15292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:29.161827 15292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:45:29.162948 15348 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:45:29.168730 15292 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:45:29.168821 15292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 14:45:29.168895 15292 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:45:29.171049 15292 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:45:29.171124 15292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 14:45:29.171427 15348 tablet_bootstrap.cc:492] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff: Bootstrap starting.
I20260619 14:45:29.182240 15292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38823
I20260619 14:45:29.183272 15292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:45:29.183604 15455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38823 every 8 connection(s)
I20260619 14:45:29.188549 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 15292
I20260619 14:45:29.189020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33265
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44835
--webserver_port=35891
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:45:29.195545 15348 log.cc:824] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:29.202898 15348 tablet_bootstrap.cc:492] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:45:29.203361 15348 tablet_bootstrap.cc:492] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff: Bootstrap complete.
I20260619 14:45:29.204476 15348 ts_tablet_manager.cc:1397] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff: Time spent bootstrapping tablet: real 0.033s user 0.017s sys 0.013s
I20260619 14:45:29.206914 15348 raft_consensus.cc:348] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260619 14:45:29.207360 15348 raft_consensus.cc:729] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d205adbdb84e52a264ef7ce665a6ff, State: Initialized, Role: FOLLOWER
I20260619 14:45:29.207907 15348 consensus_queue.cc:260] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260619 14:45:29.208900 15348 ts_tablet_manager.cc:1428] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260619 14:45:29.264946 15458 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:45:29.185955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:51873)
0619 14:45:29.186495 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:29.186565 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:29.187493 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:29.264170 (+ 76677us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":173,"threads_started":1}
W20260619 14:45:29.265564 15456 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44835 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44835: Client connection negotiation failed: client connection to 127.0.98.62:44835: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:45:29.266359 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:29.266085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:36671)
0619 14:45:29.266219 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:29.266291 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 14:45:29.267105 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:29.266890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:57073)
0619 14:45:29.267000 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:29.267059 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 14:45:29.267300 15456 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:45:29.325572 15459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:29.325866 15459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:29.325963 15459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:29.330091 15459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:29.330190 15459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:29.330288 15459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:45:29.335785 15459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35891
--tserver_master_addrs=127.0.98.62:44835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:29.336990 15459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:29.338110 15459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:29.347945 15466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:29.347918 15469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:29.348338 15459 server_base.cc:1034] running on GCE node
W20260619 14:45:29.348352 15467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:29.348831 15459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:29.349560 15459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:29.350766 15459 hybrid_clock.cc:648] HybridClock initialized: now 1781880329350729 us; error 47 us; skew 500 ppm
Jun 19 14:45:29 dist-test-slave-cr22 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:29.353780 15459 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:45:29.354822 15459 webserver.cc:460] Webserver started at http://127.0.98.2:35891/ using document root <none> and password file <none>
I20260619 14:45:29.355396 15459 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:29.355530 15459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:29.359580 15459 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:45:29.361884 15476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:29.363190 15459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:45:29.363327 15459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "1556caff2fbf42e698ccc4f464748fdc"
format_stamp: "Formatted at 2026-06-19 14:45:26 on dist-test-slave-cr22"
server_key: "bf438cc2aac8fea2d1f056f098a08f0f"
server_key_iv: "bc09a8c7d605a910eba123457dd387ca"
server_key_version: "encryptionkey@0"
I20260619 14:45:29.364015 15459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:29.398985 15459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:29.402781 15459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:29.403059 15459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:29.403889 15459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:45:29.405009 15483 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:45:29.409541 15459 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:45:29.409613 15459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 14:45:29.409693 15459 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:45:29.411533 15459 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:45:29.411582 15459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260619 14:45:29.411840 15483 tablet_bootstrap.cc:492] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc: Bootstrap starting.
I20260619 14:45:29.422633 15459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33265
I20260619 14:45:29.422724 15590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33265 every 8 connection(s)
I20260619 14:45:29.423758 15459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:45:29.427227 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 15459
I20260619 14:45:29.427347 15593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:29.425679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:48873)
0619 14:45:29.426180 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:29.426338 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":121,"threads_started":1}
I20260619 14:45:29.427750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43309
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44835
--webserver_port=40111
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:45:29.428085 15591 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44835 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44835: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
I20260619 14:45:29.428886 15593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:29.428524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:43375)
0619 14:45:29.428736 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:29.428803 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 14:45:29.429699 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:29.429307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:52561)
0619 14:45:29.429573 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:29.429626 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":96,"threads_started":1}
W20260619 14:45:29.429894 15591 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 14:45:29.430534 15483 log.cc:824] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:29.440186 15483 tablet_bootstrap.cc:492] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:45:29.440616 15483 tablet_bootstrap.cc:492] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc: Bootstrap complete.
I20260619 14:45:29.442250 15483 ts_tablet_manager.cc:1397] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc: Time spent bootstrapping tablet: real 0.031s user 0.027s sys 0.000s
I20260619 14:45:29.445859 15483 raft_consensus.cc:348] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260619 14:45:29.446409 15483 raft_consensus.cc:729] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1556caff2fbf42e698ccc4f464748fdc, State: Initialized, Role: FOLLOWER
I20260619 14:45:29.446954 15483 consensus_queue.cc:260] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260619 14:45:29.448046 15483 ts_tablet_manager.cc:1428] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260619 14:45:29.448472 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:29.448090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:45305)
0619 14:45:29.448276 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:29.448382 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 14:45:29.449136 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:29.448928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:43955)
0619 14:45:29.449036 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:29.449081 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 14:45:29.545562 15594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:29.545847 15594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:29.545925 15594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:29.550297 15594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:29.550388 15594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:29.550480 15594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:45:29.556031 15594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40111
--tserver_master_addrs=127.0.98.62:44835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:29.557123 15594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:29.558122 15594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:29.567622 15602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:29.567711 15603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:29.567711 15605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:29.567962 15594 server_base.cc:1034] running on GCE node
I20260619 14:45:29.568393 15594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:29.568992 15594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:29.570204 15594 hybrid_clock.cc:648] HybridClock initialized: now 1781880329570189 us; error 37 us; skew 500 ppm
Jun 19 14:45:29 dist-test-slave-cr22 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:29.573462 15594 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:45:29.574401 15594 webserver.cc:460] Webserver started at http://127.0.98.3:40111/ using document root <none> and password file <none>
I20260619 14:45:29.574981 15594 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:29.575052 15594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:29.579016 15594 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:45:29.581264 15612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:29.582517 15594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:45:29.582654 15594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "4ce60de966ab4295ae6d6d43f9db4b5e"
format_stamp: "Formatted at 2026-06-19 14:45:27 on dist-test-slave-cr22"
server_key: "7d935537a5f474b8c18bc4d31abc6176"
server_key_iv: "60efcef9ff761cb5665033f034da0184"
server_key_version: "encryptionkey@0"
I20260619 14:45:29.583201 15594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:29.615662 15594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:29.619134 15594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:29.619354 15594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:29.620046 15594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:45:29.620924 15594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:45:29.620970 15594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:29.621043 15594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:45:29.621075 15594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:29.630540 15594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43309
I20260619 14:45:29.630560 15725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43309 every 8 connection(s)
I20260619 14:45:29.631644 15594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:45:29.636075 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 15594
I20260619 14:45:29.636234 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:29.633908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:48205)
0619 14:45:29.634470 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:29.634591 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":125,"threads_started":1}
W20260619 14:45:29.636732 15726 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44835 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44835: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
I20260619 14:45:29.637501 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:29.637228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:34459)
0619 14:45:29.637387 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:29.637440 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 14:45:29.638151 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:29.637929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:40441)
0619 14:45:29.638059 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:29.638104 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 14:45:29.638363 15726 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 14:45:30.268376 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:30.267940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:39771)
0619 14:45:30.268208 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:30.268266 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
I20260619 14:45:30.425668 15731 raft_consensus.cc:482] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:45:30.425889 15731 raft_consensus.cc:504] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260619 14:45:30.426990 15731 leader_election.cc:288] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ce60de966ab4295ae6d6d43f9db4b5e (127.0.98.3:43309), 1556caff2fbf42e698ccc4f464748fdc (127.0.98.2:33265)
Jun 19 14:45:30 dist-test-slave-cr22 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 19 14:45:30 dist-test-slave-cr22 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260619 14:45:30.442934 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:30.427664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36305 (local address 127.0.98.3:43309)
0619 14:45:30.428046 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:30.428059 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:30.428099 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:45:30.428729 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:30.428734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:30.429091 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:30.429390 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:30.430362 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:30.431917 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:30.433890 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:30.434593 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:30.434764 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:30.438080 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:30.438088 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:30.438105 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:30.438145 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:30.440136 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:30.440716 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:30.440724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:30.440729 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:30.440887 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:30.441289 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:30.441293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:30.441295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:30.441802 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:30.441944 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:30.442464 (+ 520us) server_negotiation.cc:299] Negotiation successful
0619 14:45:30.442730 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":109,"threads_started":1}
I20260619 14:45:30.442924 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:30.427200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43309 (local address 127.0.98.1:36305)
0619 14:45:30.427716 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:30.427730 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:30.427832 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:30.429506 (+ 1674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:30.429511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:30.429748 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:30.430143 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:30.430161 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:30.432101 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:30.432103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:30.433739 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:30.433750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:30.434739 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:30.434743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:30.435039 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:30.435890 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:30.435903 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:30.437839 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:30.440266 (+ 2427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:30.440273 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:30.440286 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:30.440578 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:30.441004 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:30.441009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:30.441017 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:30.441183 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:30.441918 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:30.441924 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:30.442154 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 14:45:30.442343 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 14:45:30.442641 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":450,"spinlock_wait_cycles":4992,"thread_start_us":106,"threads_started":1}
I20260619 14:45:30.442936 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:30.427200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33265 (local address 127.0.98.1:60581)
0619 14:45:30.427407 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:30.427424 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:30.428032 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:30.429089 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:30.429095 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:30.429608 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:30.430144 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:30.430161 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:30.432099 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:30.432103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:30.433738 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:30.433750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:30.434739 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:30.434743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:30.435039 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:30.435880 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:30.435909 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:30.438207 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:30.440986 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:30.440990 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:30.440992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:30.441211 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:30.441606 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:30.441609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:30.441611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:30.441719 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:30.442328 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:30.442331 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:30.442426 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 14:45:30.442579 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 14:45:30.442720 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":217,"spinlock_wait_cycles":2048}
I20260619 14:45:30.443710 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:30.427420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60581 (local address 127.0.98.2:33265)
0619 14:45:30.427809 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:30.427824 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:30.428031 (+ 207us) server_negotiation.cc:407] Connection header received
0619 14:45:30.428311 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:30.428317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:30.428701 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:30.429020 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:30.430311 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:30.431917 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:30.433890 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:30.434593 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:30.434764 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:30.438396 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:30.438401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:30.438412 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:30.438483 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:30.440853 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:30.441341 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:30.441354 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:30.441359 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:30.441497 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:30.441827 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:30.441831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:30.441832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:30.442212 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:30.442365 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:30.442706 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 14:45:30.443559 (+ 853us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":64,"threads_started":1}
I20260619 14:45:30.444716 15545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6629348a364478aa4c155ae371a4808" candidate_uuid: "66d205adbdb84e52a264ef7ce665a6ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1556caff2fbf42e698ccc4f464748fdc" is_pre_election: true
I20260619 14:45:30.445075 15545 raft_consensus.cc:2454] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66d205adbdb84e52a264ef7ce665a6ff in term 1.
W20260619 14:45:30.445811 15343 leader_election.cc:341] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ce60de966ab4295ae6d6d43f9db4b5e (127.0.98.3:43309): Not found: Tablet not found: e6629348a364478aa4c155ae371a4808
I20260619 14:45:30.446003 15343 leader_election.cc:302] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1556caff2fbf42e698ccc4f464748fdc, 66d205adbdb84e52a264ef7ce665a6ff; no voters: 4ce60de966ab4295ae6d6d43f9db4b5e
I20260619 14:45:30.446321 15731 raft_consensus.cc:2781] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 14:45:30.446395 15731 raft_consensus.cc:482] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:45:30.446426 15731 raft_consensus.cc:3037] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:45:30.448664 15731 raft_consensus.cc:504] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260619 14:45:30.449093 15731 leader_election.cc:288] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [CANDIDATE]: Term 2 election: Requested vote from peers 4ce60de966ab4295ae6d6d43f9db4b5e (127.0.98.3:43309), 1556caff2fbf42e698ccc4f464748fdc (127.0.98.2:33265)
I20260619 14:45:30.449627 15545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6629348a364478aa4c155ae371a4808" candidate_uuid: "66d205adbdb84e52a264ef7ce665a6ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1556caff2fbf42e698ccc4f464748fdc"
I20260619 14:45:30.449790 15545 raft_consensus.cc:3037] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc [term 1 FOLLOWER]: Advancing to term 2
W20260619 14:45:30.449906 15343 leader_election.cc:341] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 4ce60de966ab4295ae6d6d43f9db4b5e (127.0.98.3:43309): Not found: Tablet not found: e6629348a364478aa4c155ae371a4808
I20260619 14:45:30.450310 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:30.449832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:42097)
0619 14:45:30.450158 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:30.450220 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":86,"threads_started":1}
I20260619 14:45:30.452220 15545 raft_consensus.cc:2454] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66d205adbdb84e52a264ef7ce665a6ff in term 2.
I20260619 14:45:30.452507 15342 leader_election.cc:302] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1556caff2fbf42e698ccc4f464748fdc, 66d205adbdb84e52a264ef7ce665a6ff; no voters: 4ce60de966ab4295ae6d6d43f9db4b5e
I20260619 14:45:30.452682 15731 raft_consensus.cc:2781] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:45:30.452880 15731 raft_consensus.cc:686] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [term 2 LEADER]: Becoming Leader. State: Replica: 66d205adbdb84e52a264ef7ce665a6ff, State: Running, Role: LEADER
I20260619 14:45:30.453248 15731 consensus_queue.cc:237] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260619 14:45:30.453626 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:30.453310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:55513)
0619 14:45:30.453511 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:30.453560 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 14:45:30.639598 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:30.638964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:37343)
0619 14:45:30.639295 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:30.639479 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":107,"threads_started":1}
I20260619 14:45:30.864764 15545 raft_consensus.cc:1261] T e6629348a364478aa4c155ae371a4808 P 1556caff2fbf42e698ccc4f464748fdc [term 2 FOLLOWER]: Refusing update from remote peer 66d205adbdb84e52a264ef7ce665a6ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260619 14:45:30.865480 15731 consensus_queue.cc:1035] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260619 14:45:30.867403 15343 consensus_peers.cc:487] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff -> Peer 4ce60de966ab4295ae6d6d43f9db4b5e (127.0.98.3:43309): Couldn't send request to peer 4ce60de966ab4295ae6d6d43f9db4b5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6629348a364478aa4c155ae371a4808. This is attempt 1: this message will repeat every 5th retry.
I20260619 14:45:30.867903 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:30.867344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:53789)
0619 14:45:30.867742 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:30.867808 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260619 14:45:30.870173 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:30.869895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:40529)
0619 14:45:30.870025 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:30.870096 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 14:45:30.870613 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:30.870410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:56345)
0619 14:45:30.870502 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:30.870560 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 14:45:31.475682 15744 ts_tablet_manager.cc:927] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: Initiating tablet copy from peer 66d205adbdb84e52a264ef7ce665a6ff (127.0.98.1:38823)
I20260619 14:45:31.476305 15744 tablet_copy_client.cc:316] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:38823
Jun 19 14:45:31 dist-test-slave-cr22 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:45:31.487586 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:31.476903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38823 (local address 127.0.98.3:53559)
0619 14:45:31.477203 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:31.477225 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:31.477383 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:31.477862 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:31.477867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:31.478240 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:31.478351 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:31.478357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:31.479427 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:31.479429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:31.480225 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:31.480234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:31.480912 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:31.480915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:31.481022 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:31.481621 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:31.481634 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:31.483262 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:31.485658 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:31.485666 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:31.485672 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:31.485944 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:31.486271 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:31.486273 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:31.486275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:31.486360 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:31.486839 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:31.486846 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:31.486981 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 14:45:31.487125 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 14:45:31.487361 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":100,"threads_started":1}
I20260619 14:45:31.487782 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:31.477037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53559 (local address 127.0.98.1:38823)
0619 14:45:31.477373 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:31.477379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:31.477403 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:45:31.477523 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:31.477529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:31.477710 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:31.477834 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:31.478486 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:31.479311 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:31.480349 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:31.480729 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:31.480821 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:31.483423 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:31.483432 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:31.483506 (+ 74us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:31.483541 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:31.485525 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:31.486067 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:31.486072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:31.486073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:31.486160 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:31.486449 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:31.486451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:31.486453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:31.486728 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:31.486837 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:31.487241 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 14:45:31.487499 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":70,"threads_started":1}
I20260619 14:45:31.489051 15412 tablet_copy_service.cc:140] P 66d205adbdb84e52a264ef7ce665a6ff: Received BeginTabletCopySession request for tablet e6629348a364478aa4c155ae371a4808 from peer 4ce60de966ab4295ae6d6d43f9db4b5e ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53559)
I20260619 14:45:31.489231 15412 tablet_copy_service.cc:161] P 66d205adbdb84e52a264ef7ce665a6ff: Beginning new tablet copy session on tablet e6629348a364478aa4c155ae371a4808 from peer 4ce60de966ab4295ae6d6d43f9db4b5e at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53559: session id = 4ce60de966ab4295ae6d6d43f9db4b5e-e6629348a364478aa4c155ae371a4808
I20260619 14:45:31.490360 15412 tablet_copy_source_session.cc:205] T e6629348a364478aa4c155ae371a4808 P 66d205adbdb84e52a264ef7ce665a6ff: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 14:45:31.494295 15744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6629348a364478aa4c155ae371a4808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:31.501132 15744 tablet_copy_client.cc:791] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: tablet copy: Starting download of 0 data blocks...
I20260619 14:45:31.501390 15744 tablet_copy_client.cc:655] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: tablet copy: Starting download of 1 WAL segments...
I20260619 14:45:31.503248 15744 tablet_copy_client.cc:531] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 14:45:31.505162 15744 tablet_bootstrap.cc:492] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: Bootstrap starting.
I20260619 14:45:31.510528 15744 log.cc:824] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:31.517107 15744 tablet_bootstrap.cc:492] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:45:31.517426 15744 tablet_bootstrap.cc:492] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: Bootstrap complete.
I20260619 14:45:31.517792 15744 ts_tablet_manager.cc:1397] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20260619 14:45:31.520248 15744 raft_consensus.cc:348] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260619 14:45:31.520490 15744 raft_consensus.cc:729] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce60de966ab4295ae6d6d43f9db4b5e, State: Initialized, Role: FOLLOWER
I20260619 14:45:31.520901 15744 consensus_queue.cc:260] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce60de966ab4295ae6d6d43f9db4b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } } peers { permanent_uuid: "1556caff2fbf42e698ccc4f464748fdc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33265 } } peers { permanent_uuid: "66d205adbdb84e52a264ef7ce665a6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260619 14:45:31.521790 15744 ts_tablet_manager.cc:1428] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260619 14:45:31.522085 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:31.521792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:44333)
0619 14:45:31.521932 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:31.522016 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 14:45:31.522552 15412 tablet_copy_service.cc:342] P 66d205adbdb84e52a264ef7ce665a6ff: Request end of tablet copy session 4ce60de966ab4295ae6d6d43f9db4b5e-e6629348a364478aa4c155ae371a4808 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53559
I20260619 14:45:31.522673 15412 tablet_copy_service.cc:433] P 66d205adbdb84e52a264ef7ce665a6ff: ending tablet copy session 4ce60de966ab4295ae6d6d43f9db4b5e-e6629348a364478aa4c155ae371a4808 on tablet e6629348a364478aa4c155ae371a4808 with peer 4ce60de966ab4295ae6d6d43f9db4b5e
I20260619 14:45:31.522811 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:31.522547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:54045)
0619 14:45:31.522657 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:31.522707 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 14:45:31.871261 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:31.870859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:41069)
0619 14:45:31.871110 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:31.871165 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
I20260619 14:45:31.871733 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:31.871289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:40639)
0619 14:45:31.871610 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:31.871659 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":117,"threads_started":1}
I20260619 14:45:32.083861 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:32.083392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:34429)
0619 14:45:32.083699 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:32.083767 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216}
I20260619 14:45:32.084064 15680 raft_consensus.cc:1203] T e6629348a364478aa4c155ae371a4808 P 4ce60de966ab4295ae6d6d43f9db4b5e [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260619 14:45:32.085279 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:32.084911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:55461)
0619 14:45:32.085152 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:32.085205 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
I20260619 14:45:32.873028 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:32.872552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:36009)
0619 14:45:32.872859 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:32.872921 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":78,"threads_started":1}
I20260619 14:45:33.085242 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:33.084783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:53835)
0619 14:45:33.085080 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:33.085143 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":80,"threads_started":1}
I20260619 14:45:33.086535 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:33.086081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:36343)
0619 14:45:33.086348 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:33.086428 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
I20260619 14:45:33.874588 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:33.874087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:44969)
0619 14:45:33.874395 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:33.874458 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":80,"threads_started":1}
I20260619 14:45:34.086679 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:34.086134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:39521)
0619 14:45:34.086498 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:34.086579 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":78,"threads_started":1}
I20260619 14:45:34.087909 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:34.087340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:35767)
0619 14:45:34.087719 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:34.087808 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":142,"threads_started":1}
I20260619 14:45:34.876331 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:34.875817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:52845)
0619 14:45:34.876114 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:34.876200 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":114,"threads_started":1}
I20260619 14:45:35.088212 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:35.087798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:56803)
0619 14:45:35.088047 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:35.088108 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
I20260619 14:45:35.088992 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:35.088761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:43003)
0619 14:45:35.088887 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:35.088937 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 14:45:35.089516 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:35.089045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:34521)
0619 14:45:35.089372 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:35.089425 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":93,"threads_started":1}
I20260619 14:45:35.877714 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:35.877251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:36505)
0619 14:45:35.877548 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:35.877614 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
I20260619 14:45:36.090164 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:36.089756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:33919)
0619 14:45:36.090018 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:36.090071 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
I20260619 14:45:36.090562 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:36.090154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:50437)
0619 14:45:36.090427 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:36.090482 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":82,"threads_started":1}
I20260619 14:45:36.879108 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:36.878641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:34565)
0619 14:45:36.878936 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:36.879000 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
I20260619 14:45:37.091549 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:37.091063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:58369)
0619 14:45:37.091335 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:37.091393 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":101,"threads_started":1}
I20260619 14:45:37.091786 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:37.091301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:59117)
0619 14:45:37.091598 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:37.091667 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":259,"threads_started":1}
I20260619 14:45:37.880434 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:37.879996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:33955)
0619 14:45:37.880270 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:37.880334 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":73,"threads_started":1}
I20260619 14:45:38.092993 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:38.092542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:37237)
0619 14:45:38.092824 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:38.092888 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":119,"threads_started":1}
I20260619 14:45:38.093351 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:38.092618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:36389)
0619 14:45:38.093147 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:38.093220 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":370,"threads_started":1}
I20260619 14:45:38.881743 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:38.881244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:52889)
0619 14:45:38.881554 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:38.881625 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":78,"threads_started":1}
I20260619 14:45:39.094615 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:39.094081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:60483)
0619 14:45:39.094370 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:39.094451 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":106,"threads_started":1}
I20260619 14:45:39.094915 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:39.094359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:59961)
0619 14:45:39.094732 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:39.094800 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":104,"threads_started":1}
I20260619 14:45:39.883256 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:39.882792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:40751)
0619 14:45:39.883095 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:39.883158 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":91,"threads_started":1}
I20260619 14:45:40.096268 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:40.095794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:47499)
0619 14:45:40.096103 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:40.096168 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":97,"threads_started":1}
I20260619 14:45:40.096385 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:40.095845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:47079)
0619 14:45:40.096193 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:40.096255 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":62,"threads_started":1}
I20260619 14:45:40.884655 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:40.884102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:39521)
0619 14:45:40.884396 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:40.884476 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":79,"threads_started":1}
I20260619 14:45:41.097672 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:41.097242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:54911)
0619 14:45:41.097487 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:41.097558 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":94,"threads_started":1}
I20260619 14:45:41.097684 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:41.097169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:41399)
0619 14:45:41.097457 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:41.097578 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":114,"threads_started":1}
I20260619 14:45:41.885878 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:41.885429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:48967)
0619 14:45:41.885733 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:41.885787 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":123,"threads_started":1}
I20260619 14:45:42.099086 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:42.098653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:40581)
0619 14:45:42.098917 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:42.098990 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":106,"threads_started":1}
I20260619 14:45:42.099149 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:42.098647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:50853)
0619 14:45:42.098984 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:42.099032 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":112,"threads_started":1}
I20260619 14:45:42.887390 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:42.886800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:49727)
0619 14:45:42.887166 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:42.887256 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":88,"threads_started":1}
I20260619 14:45:43.100481 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:43.100016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:37219)
0619 14:45:43.100347 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:43.100399 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":94,"threads_started":1}
I20260619 14:45:43.100472 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:43.100016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:50761)
0619 14:45:43.100314 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:43.100374 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":82,"threads_started":1}
I20260619 14:45:43.888667 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:43.888218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.2:42161)
0619 14:45:43.888503 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:43.888566 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":75,"threads_started":1}
I20260619 14:45:44.101926 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:44.101446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.3:34749)
0619 14:45:44.101761 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:44.101825 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":84,"threads_started":1}
I20260619 14:45:44.101948 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:44.101449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44835 (local address 127.0.98.1:51965)
0619 14:45:44.101804 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:44.101861 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
I20260619 14:45:44.648746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 15292
I20260619 14:45:44.655010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 15459
I20260619 14:45:44.660806 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 15594
2026-06-19T14:45:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (20089 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:45:45 dist-test-slave-cr22 krb5kdc[15796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:45:45 dist-test-slave-cr22 krb5kdc[15796](info): set up 2 sockets
Jun 19 14:45:45 dist-test-slave-cr22 krb5kdc[15796](info): commencing operation
krb5kdc: starting...
W20260619 14:45:46.163866 392 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:45:46 dist-test-slave-cr22 krb5kdc[15796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:45:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:45:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:45:46.294719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45837
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:45:46.406095 15809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:46.406378 15809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:46.406420 15809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:46.410517 15809 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 14:45:46.410583 15809 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 14:45:46.410598 15809 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 14:45:46.410614 15809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:45:46.410633 15809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:46.410651 15809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:46.410713 15809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:45:46.410733 15809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:45:46.416082 15809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:46.417178 15809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:46.418198 15809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:46.427562 15814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:46.427691 15809 server_base.cc:1034] running on GCE node
W20260619 14:45:46.427699 15815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:46.427716 15817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:46.428495 15809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:46.429647 15809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:46.430877 15809 hybrid_clock.cc:648] HybridClock initialized: now 1781880346430847 us; error 44 us; skew 500 ppm
Jun 19 14:45:46 dist-test-slave-cr22 krb5kdc[15796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:46.433889 15809 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:45:46.434825 15809 webserver.cc:460] Webserver started at http://127.0.98.62:46491/ using document root <none> and password file <none>
I20260619 14:45:46.435335 15809 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:46.435393 15809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:46.435686 15809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:45:46.437707 15809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "1fea269cd46749f88370613b756334a2"
format_stamp: "Formatted at 2026-06-19 14:45:46 on dist-test-slave-cr22"
server_key: "36403746de8bf71fb0570e6de8c709ec"
server_key_iv: "db02aae0cd6eec9b2ac4cd9d391b0227"
server_key_version: "encryptionkey@0"
I20260619 14:45:46.438210 15809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "1fea269cd46749f88370613b756334a2"
format_stamp: "Formatted at 2026-06-19 14:45:46 on dist-test-slave-cr22"
server_key: "36403746de8bf71fb0570e6de8c709ec"
server_key_iv: "db02aae0cd6eec9b2ac4cd9d391b0227"
server_key_version: "encryptionkey@0"
I20260619 14:45:46.441991 15809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 14:45:46.444438 15824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:46.445711 15809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:45:46.445813 15809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "1fea269cd46749f88370613b756334a2"
format_stamp: "Formatted at 2026-06-19 14:45:46 on dist-test-slave-cr22"
server_key: "36403746de8bf71fb0570e6de8c709ec"
server_key_iv: "db02aae0cd6eec9b2ac4cd9d391b0227"
server_key_version: "encryptionkey@0"
I20260619 14:45:46.445963 15809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:46.472970 15809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:46.476383 15809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:46.476621 15809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:46.484081 15809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45837
I20260619 14:45:46.484361 15876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45837 every 8 connection(s)
I20260619 14:45:46.485065 15809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:45:46.488360 15877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:46.492009 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 15809
I20260619 14:45:46.492142 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:45:46.492444 392 external_mini_cluster.cc:1428] Setting key 1c6a1d6cf4a1dd359a7d2447c2ed23c6
I20260619 14:45:46.494351 15877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: Bootstrap starting.
I20260619 14:45:46.497165 15877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: Neither blocks nor log segments found. Creating new log.
I20260619 14:45:46.498104 15877 log.cc:824] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:46.500653 15877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: No bootstrap required, opened a new log
Jun 19 14:45:46 dist-test-slave-cr22 krb5kdc[15796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:45:46.504784 15877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fea269cd46749f88370613b756334a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45837 } }
I20260619 14:45:46.505120 15877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:45:46.505172 15877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fea269cd46749f88370613b756334a2, State: Initialized, Role: FOLLOWER
I20260619 14:45:46.505697 15877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fea269cd46749f88370613b756334a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45837 } }
I20260619 14:45:46.505872 15877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:45:46.505944 15877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:45:46.506039 15877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:45:46.507301 15877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fea269cd46749f88370613b756334a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45837 } }
I20260619 14:45:46.507725 15877 leader_election.cc:302] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fea269cd46749f88370613b756334a2; no voters:
I20260619 14:45:46.508121 15877 leader_election.cc:288] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:45:46.508260 15882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:45:46.508667 15882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 1 LEADER]: Becoming Leader. State: Replica: 1fea269cd46749f88370613b756334a2, State: Running, Role: LEADER
I20260619 14:45:46.509056 15882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fea269cd46749f88370613b756334a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45837 } }
I20260619 14:45:46.509243 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:46.494008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59032 (local address 127.0.98.62:45837)
0619 14:45:46.494436 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:46.494455 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:46.494497 (+ 42us) server_negotiation.cc:407] Connection header received
0619 14:45:46.495322 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:46.495344 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:46.495797 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:46.496158 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:46.497151 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:46.498745 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:46.499840 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:46.500518 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:46.500699 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:46.503621 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:46.503628 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:46.503648 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:46.503681 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:46.505728 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:46.506341 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:46.506347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:46.506353 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:46.506496 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:46.506815 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:46.506818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:46.506820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:46.507378 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:46.507604 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:46.507942 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 14:45:46.508286 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":115,"threads_started":1}
I20260619 14:45:46.509521 15877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:45:46.512153 15883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fea269cd46749f88370613b756334a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fea269cd46749f88370613b756334a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45837 } } }
I20260619 14:45:46.512313 15883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [sys.catalog]: This master's current role is: LEADER
I20260619 14:45:46.512683 15884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fea269cd46749f88370613b756334a2. Latest consensus state: current_term: 1 leader_uuid: "1fea269cd46749f88370613b756334a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fea269cd46749f88370613b756334a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45837 } } }
I20260619 14:45:46.512782 15884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [sys.catalog]: This master's current role is: LEADER
W20260619 14:45:46.514127 15896 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:45:46.514189 15896 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:45:46.514278 15897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:45:46.515002 15897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:45:46.520992 15897 catalog_manager.cc:1347] Generated new cluster ID: e5217a81c1d341539d09a86632200d86
I20260619 14:45:46.521070 15897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:45:46.529160 15897 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:45:46.530407 15897 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:45:46.536921 15897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: Generated new TSK 0
I20260619 14:45:46.537653 15897 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:45:46.575012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45837
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:45:46.691160 15903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:46.691496 15903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:46.691608 15903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:46.695672 15903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:46.695755 15903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:46.695842 15903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:45:46.701406 15903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:46.702527 15903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:46.703627 15903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:46.714032 15908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:46.714093 15911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:46.714203 15903 server_base.cc:1034] running on GCE node
W20260619 14:45:46.714032 15909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:46.714783 15903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:46.715639 15903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:46.716858 15903 hybrid_clock.cc:648] HybridClock initialized: now 1781880346716844 us; error 35 us; skew 500 ppm
Jun 19 14:45:46 dist-test-slave-cr22 krb5kdc[15796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:46.720099 15903 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:45:46.721093 15903 webserver.cc:460] Webserver started at http://127.0.98.1:37193/ using document root <none> and password file <none>
I20260619 14:45:46.721693 15903 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:46.721786 15903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:46.722028 15903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:45:46.724222 15903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "8504541fa3b84b0da8e0376ff83c6040"
format_stamp: "Formatted at 2026-06-19 14:45:46 on dist-test-slave-cr22"
server_key: "b7b114f20f265bbc27b5dacd35bde55e"
server_key_iv: "0db6786587a3a9b4dd80179df0cd0394"
server_key_version: "encryptionkey@0"
I20260619 14:45:46.724817 15903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8504541fa3b84b0da8e0376ff83c6040"
format_stamp: "Formatted at 2026-06-19 14:45:46 on dist-test-slave-cr22"
server_key: "b7b114f20f265bbc27b5dacd35bde55e"
server_key_iv: "0db6786587a3a9b4dd80179df0cd0394"
server_key_version: "encryptionkey@0"
I20260619 14:45:46.728830 15903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 14:45:46.731662 15918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:46.733047 15903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:45:46.733224 15903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "8504541fa3b84b0da8e0376ff83c6040"
format_stamp: "Formatted at 2026-06-19 14:45:46 on dist-test-slave-cr22"
server_key: "b7b114f20f265bbc27b5dacd35bde55e"
server_key_iv: "0db6786587a3a9b4dd80179df0cd0394"
server_key_version: "encryptionkey@0"
I20260619 14:45:46.733366 15903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:46.742782 15903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:46.745843 15903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:46.746060 15903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:46.746644 15903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:45:46.747486 15903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:45:46.747563 15903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:46.747629 15903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:45:46.747682 15903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:46.757654 15903 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45829
I20260619 14:45:46.757676 16031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45829 every 8 connection(s)
I20260619 14:45:46.758746 15903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:45:46.762218 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 15903
I20260619 14:45:46.762395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:45:46.762735 392 external_mini_cluster.cc:1428] Setting key 9d9b3ed8250c71960d9ff0e71f97cf74
Jun 19 14:45:46 dist-test-slave-cr22 krb5kdc[15796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:45:46.775835 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:46.760771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42321 (local address 127.0.98.62:45837)
0619 14:45:46.760921 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:46.760926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:46.762425 (+ 1499us) server_negotiation.cc:407] Connection header received
0619 14:45:46.763654 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:46.763657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:46.763701 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:46.763763 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:46.765386 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:46.765932 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:46.767553 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:46.767989 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:46.768117 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:46.771141 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:46.771147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:46.771151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:46.771190 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:46.772983 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:46.773756 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:46.773759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:46.773760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:46.773852 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:46.774602 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:46.774607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:46.774610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:46.774825 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:46.774942 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:46.775528 (+ 586us) server_negotiation.cc:299] Negotiation successful
0619 14:45:46.775667 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:45:46.776942 16034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:46.761207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:42321)
0619 14:45:46.762196 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:46.762260 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:46.763357 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:46.763941 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:46.763950 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:46.764450 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:46.765191 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:46.765215 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:46.766361 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:46.766369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:46.767377 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:46.767387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:46.768181 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:46.768185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:46.768350 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:46.769055 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:46.769072 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:46.770932 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:46.773124 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:46.773129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:46.773142 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:46.773594 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:46.773988 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:46.773991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:46.773992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:46.774379 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:46.774953 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:46.774960 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:46.775246 (+ 286us) client_negotiation.cc:769] Sending connection context
0619 14:45:46.775480 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 14:45:46.775839 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":111,"threads_started":1}
I20260619 14:45:46.778395 16032 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45837
I20260619 14:45:46.778687 16032 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:46.779417 16032 heartbeater.cc:511] Master 127.0.98.62:45837 requested a full tablet report, sending...
I20260619 14:45:46.781344 15841 ts_manager.cc:194] Registered new tserver with Master: 8504541fa3b84b0da8e0376ff83c6040 (127.0.98.1:45829)
I20260619 14:45:46.782598 15841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:45:46.795351 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45837
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:45:46.914817 16037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:46.915138 16037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:46.915217 16037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:46.919597 16037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:46.919716 16037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:46.919822 16037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:45:46.925544 16037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:46.926726 16037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:46.927810 16037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:46.938428 16043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:46.938408 16045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:46.938412 16042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:46.938870 16037 server_base.cc:1034] running on GCE node
I20260619 14:45:46.939284 16037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:46.940068 16037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:46.941267 16037 hybrid_clock.cc:648] HybridClock initialized: now 1781880346941247 us; error 48 us; skew 500 ppm
Jun 19 14:45:46 dist-test-slave-cr22 krb5kdc[15796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:46.944293 16037 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:45:46.945281 16037 webserver.cc:460] Webserver started at http://127.0.98.2:41755/ using document root <none> and password file <none>
I20260619 14:45:46.945917 16037 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:46.946022 16037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:46.946269 16037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:45:46.948536 16037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "411bce22959b49a88031e65d983d5244"
format_stamp: "Formatted at 2026-06-19 14:45:46 on dist-test-slave-cr22"
server_key: "9db852459ecdfdf8da62abe158911c92"
server_key_iv: "159bcfcd24e32b3d804007c0c93f63bb"
server_key_version: "encryptionkey@0"
I20260619 14:45:46.949148 16037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "411bce22959b49a88031e65d983d5244"
format_stamp: "Formatted at 2026-06-19 14:45:46 on dist-test-slave-cr22"
server_key: "9db852459ecdfdf8da62abe158911c92"
server_key_iv: "159bcfcd24e32b3d804007c0c93f63bb"
server_key_version: "encryptionkey@0"
I20260619 14:45:46.953392 16037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:45:46.956033 16052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:46.957320 16037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:45:46.957476 16037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "411bce22959b49a88031e65d983d5244"
format_stamp: "Formatted at 2026-06-19 14:45:46 on dist-test-slave-cr22"
server_key: "9db852459ecdfdf8da62abe158911c92"
server_key_iv: "159bcfcd24e32b3d804007c0c93f63bb"
server_key_version: "encryptionkey@0"
I20260619 14:45:46.957639 16037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:46.985386 16037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:46.989166 16037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:46.989401 16037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:46.990044 16037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:45:46.990981 16037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:45:46.991058 16037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:46.991135 16037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:45:46.991191 16037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:47.000819 16037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39519
I20260619 14:45:47.000964 16165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39519 every 8 connection(s)
I20260619 14:45:47.001832 16037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:45:47.003240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 16037
I20260619 14:45:47.003400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:45:47.003758 392 external_mini_cluster.cc:1428] Setting key b792786fb4e7d7d2f04881cb72bb36b8
Jun 19 14:45:47 dist-test-slave-cr22 krb5kdc[15796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:45:47.018831 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:47.004170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49123 (local address 127.0.98.62:45837)
0619 14:45:47.004328 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:47.004338 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:47.005795 (+ 1457us) server_negotiation.cc:407] Connection header received
0619 14:45:47.007139 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:47.007142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:47.007188 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:47.007269 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:47.008724 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.009304 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.011328 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.011674 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.011787 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:47.014974 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:47.014978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:47.014981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:47.015012 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:47.016592 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:47.017211 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:47.017215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:47.017217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:47.017300 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:47.017655 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:47.017658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:47.017660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:47.017850 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:47.017957 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:47.018514 (+ 557us) server_negotiation.cc:299] Negotiation successful
0619 14:45:47.018655 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 14:45:47.019726 16168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:47.004721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:49123)
0619 14:45:47.005623 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:47.005682 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:47.006898 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:47.007573 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:47.007584 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:47.008033 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:47.008546 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:47.008560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.009561 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.009567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:47.011149 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:47.011164 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.011856 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.011863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:47.012139 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:47.012978 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:47.013002 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:47.014796 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:47.016812 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:47.016818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:47.016835 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:47.017106 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:47.017395 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:47.017398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:47.017400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:47.017550 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:47.017956 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:47.017963 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:47.018232 (+ 269us) client_negotiation.cc:769] Sending connection context
0619 14:45:47.018457 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 14:45:47.018793 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":163,"threads_started":1}
I20260619 14:45:47.021062 16166 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45837
I20260619 14:45:47.021322 16166 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:47.021835 16166 heartbeater.cc:511] Master 127.0.98.62:45837 requested a full tablet report, sending...
I20260619 14:45:47.023147 15841 ts_manager.cc:194] Registered new tserver with Master: 411bce22959b49a88031e65d983d5244 (127.0.98.2:39519)
I20260619 14:45:47.024091 15841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:45:47.035302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45837
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:45:47.154933 16171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:45:47.155210 16171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:45:47.155256 16171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:45:47.159300 16171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:45:47.159365 16171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:45:47.159478 16171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:45:47.164950 16171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:45:47.166110 16171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:45:47.167135 16171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:45:47.176755 16177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:47.176777 16176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:45:47.176774 16179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:45:47.177124 16171 server_base.cc:1034] running on GCE node
I20260619 14:45:47.177500 16171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:45:47.178082 16171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:45:47.179252 16171 hybrid_clock.cc:648] HybridClock initialized: now 1781880347179225 us; error 39 us; skew 500 ppm
Jun 19 14:45:47 dist-test-slave-cr22 krb5kdc[15796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:45:47.182057 16171 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:45:47.182909 16171 webserver.cc:460] Webserver started at http://127.0.98.3:41801/ using document root <none> and password file <none>
I20260619 14:45:47.183414 16171 fs_manager.cc:359] Metadata directory not provided
I20260619 14:45:47.183542 16171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:45:47.183717 16171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:45:47.185689 16171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "e416f9f3d8104e42826ed1005c478b30"
format_stamp: "Formatted at 2026-06-19 14:45:47 on dist-test-slave-cr22"
server_key: "7fa2191f19e9ad0a012a4e6ba7085737"
server_key_iv: "c1933ecfc8e74392b51c5ab5fb9010e6"
server_key_version: "encryptionkey@0"
I20260619 14:45:47.186193 16171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e416f9f3d8104e42826ed1005c478b30"
format_stamp: "Formatted at 2026-06-19 14:45:47 on dist-test-slave-cr22"
server_key: "7fa2191f19e9ad0a012a4e6ba7085737"
server_key_iv: "c1933ecfc8e74392b51c5ab5fb9010e6"
server_key_version: "encryptionkey@0"
I20260619 14:45:47.189873 16171 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 14:45:47.192301 16186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:47.193635 16171 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:45:47.193729 16171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "e416f9f3d8104e42826ed1005c478b30"
format_stamp: "Formatted at 2026-06-19 14:45:47 on dist-test-slave-cr22"
server_key: "7fa2191f19e9ad0a012a4e6ba7085737"
server_key_iv: "c1933ecfc8e74392b51c5ab5fb9010e6"
server_key_version: "encryptionkey@0"
I20260619 14:45:47.193861 16171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:45:47.219002 16171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:45:47.222438 16171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:45:47.222654 16171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:45:47.223272 16171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:45:47.224265 16171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:45:47.224344 16171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:47.224404 16171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:45:47.224457 16171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:45:47.234146 16171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35975
I20260619 14:45:47.234170 16299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35975 every 8 connection(s)
I20260619 14:45:47.235075 16171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:45:47.242792 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 16171
I20260619 14:45:47.242951 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:45:47.243284 392 external_mini_cluster.cc:1428] Setting key 5588333533c387202b0064418d227d1d
Jun 19 14:45:47 dist-test-slave-cr22 krb5kdc[15796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:45:47.250214 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:47.237376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52691 (local address 127.0.98.62:45837)
0619 14:45:47.237590 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:47.237597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:47.238534 (+ 937us) server_negotiation.cc:407] Connection header received
0619 14:45:47.239353 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:47.239356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:47.239406 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:47.239544 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:47.240644 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.241317 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.242568 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.242951 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.243081 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:47.246299 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:47.246303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:47.246305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:47.246337 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:47.247934 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:47.248478 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:47.248481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:47.248482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:47.248619 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:47.248940 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:47.248942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:47.248944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:47.249155 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:47.249239 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:47.249936 (+ 697us) server_negotiation.cc:299] Negotiation successful
0619 14:45:47.250075 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 14:45:47.250976 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:47.237726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.3:52691)
0619 14:45:47.238384 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:47.238423 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:47.239158 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:47.239673 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:47.239680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:47.240025 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:45:47.240494 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:47.240507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.241495 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.241498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:47.242443 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:47.242453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.243137 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.243140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:47.243288 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:47.244066 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:45:47.244099 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:45:47.246108 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:45:47.248063 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:47.248068 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:47.248080 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:47.248367 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:47.248713 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:45:47.248715 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:45:47.248716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:45:47.248845 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:45:47.249254 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:45:47.249259 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:45:47.249660 (+ 401us) client_negotiation.cc:769] Sending connection context
0619 14:45:47.249825 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 14:45:47.250105 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":123,"threads_started":1}
I20260619 14:45:47.252334 16300 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45837
I20260619 14:45:47.252586 16300 heartbeater.cc:464] Registering TS with master...
I20260619 14:45:47.253203 16300 heartbeater.cc:511] Master 127.0.98.62:45837 requested a full tablet report, sending...
I20260619 14:45:47.254304 15841 ts_manager.cc:194] Registered new tserver with Master: e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975)
I20260619 14:45:47.255028 15841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52691
I20260619 14:45:47.257699 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:45:47.268314 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:47.259971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59040 (local address 127.0.98.62:45837)
0619 14:45:47.260093 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:47.260097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:47.260269 (+ 172us) server_negotiation.cc:407] Connection header received
0619 14:45:47.260425 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:47.260428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:47.260470 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:47.260546 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:45:47.261374 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.261942 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.262933 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.263152 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.263243 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:47.264746 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:45:47.264750 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:45:47.264752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:45:47.264775 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:45:47.266507 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:47.266944 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:47.266946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:47.266948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:47.267026 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:45:47.267318 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:45:47.267377 (+ 59us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:45:47.267378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:45:47.267684 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:45:47.267802 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:47.268051 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 14:45:47.268155 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:45:47.271504 15841 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:45:47.274557 15841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:45:47.289492 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:47.283615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58744 (local address 127.0.98.2:39519)
0619 14:45:47.284749 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:47.284762 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:47.284787 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:45:47.284866 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:47.284871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:47.285064 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:47.285349 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:47.285583 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.286404 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.288217 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.288785 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.288951 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:47.289105 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:47.289252 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 14:45:47.289340 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1036,"thread_start_us":113,"threads_started":1}
I20260619 14:45:47.289495 16314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:47.283396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39519 (local address 127.0.0.1:58744)
0619 14:45:47.284262 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:47.284318 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:47.284447 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:47.285234 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:47.285237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:47.285249 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:47.285456 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:47.285466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.286547 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.286553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:47.288100 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:47.288109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.288949 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.288952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:47.289056 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:47.289092 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:45:47.289215 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 14:45:47.289335 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":168,"thread_start_us":131,"threads_started":1}
I20260619 14:45:47.289927 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:47.283826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45829 (local address 127.0.0.1:59676)
0619 14:45:47.284655 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:47.284669 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:47.284738 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:47.285412 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:47.285415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:47.285428 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:47.285555 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:47.285564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.286590 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.286596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:47.288120 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:47.288128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.289201 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.289208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:47.289304 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:47.289328 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:45:47.289374 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 14:45:47.289438 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":46,"thread_start_us":94,"threads_started":1}
I20260619 14:45:47.290002 16316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:47.284209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.0.98.1:45829)
0619 14:45:47.285028 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:47.285040 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:47.285062 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:45:47.285133 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:47.285138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:47.285297 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:47.285443 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:47.285669 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.286450 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.288255 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.289062 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.289567 (+ 505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:47.289695 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:47.289741 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 14:45:47.289822 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":110,"threads_started":1}
I20260619 14:45:47.291113 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:47.283654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35975 (local address 127.0.0.1:41778)
0619 14:45:47.284390 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:47.284404 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:47.284508 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:47.285153 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:47.285162 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:47.285191 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:47.286562 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:47.286574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.288008 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.288011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:47.289152 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:47.289165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.290696 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.290700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:47.290792 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:47.290814 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:45:47.290888 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:45:47.290959 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":1151,"thread_start_us":83,"threads_started":1}
I20260619 14:45:47.291620 16315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:47.284197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41778 (local address 127.0.98.3:35975)
0619 14:45:47.284640 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:47.284652 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:47.284677 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:45:47.284755 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:47.284764 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:47.284981 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:47.285156 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:47.286686 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.287872 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.289283 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.290093 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.290215 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:47.290632 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:47.291312 (+ 680us) server_negotiation.cc:299] Negotiation successful
0619 14:45:47.291406 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":129,"threads_started":1}
I20260619 14:45:47.292958 16100 tablet_service.cc:1467] Processing CreateTablet for tablet dcc6e2275f4c4785a81958f4a504b431 (DEFAULT_TABLE table=test-table [id=ea5aa4bb2cb54c7ca82618dee1aaf07b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:45:47.293263 15966 tablet_service.cc:1467] Processing CreateTablet for tablet dcc6e2275f4c4785a81958f4a504b431 (DEFAULT_TABLE table=test-table [id=ea5aa4bb2cb54c7ca82618dee1aaf07b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:45:47.294121 16100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc6e2275f4c4785a81958f4a504b431. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:47.294226 15966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc6e2275f4c4785a81958f4a504b431. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:47.294111 16234 tablet_service.cc:1467] Processing CreateTablet for tablet dcc6e2275f4c4785a81958f4a504b431 (DEFAULT_TABLE table=test-table [id=ea5aa4bb2cb54c7ca82618dee1aaf07b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:45:47.295024 16234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc6e2275f4c4785a81958f4a504b431. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:45:47.300639 16317 tablet_bootstrap.cc:492] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040: Bootstrap starting.
I20260619 14:45:47.302244 16318 tablet_bootstrap.cc:492] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244: Bootstrap starting.
I20260619 14:45:47.302278 16319 tablet_bootstrap.cc:492] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30: Bootstrap starting.
I20260619 14:45:47.304106 16319 tablet_bootstrap.cc:654] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30: Neither blocks nor log segments found. Creating new log.
I20260619 14:45:47.304107 16317 tablet_bootstrap.cc:654] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040: Neither blocks nor log segments found. Creating new log.
I20260619 14:45:47.304442 16318 tablet_bootstrap.cc:654] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244: Neither blocks nor log segments found. Creating new log.
I20260619 14:45:47.305095 16318 log.cc:824] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:47.305083 16317 log.cc:824] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:47.305213 16319 log.cc:824] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30: Log is configured to *not* fsync() on all Append() calls
I20260619 14:45:47.306941 16318 tablet_bootstrap.cc:492] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244: No bootstrap required, opened a new log
I20260619 14:45:47.307068 16317 tablet_bootstrap.cc:492] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040: No bootstrap required, opened a new log
I20260619 14:45:47.307196 16318 ts_tablet_manager.cc:1397] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:45:47.307248 16317 ts_tablet_manager.cc:1397] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 14:45:47.307271 16319 tablet_bootstrap.cc:492] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30: No bootstrap required, opened a new log
I20260619 14:45:47.307425 16319 ts_tablet_manager.cc:1397] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:45:47.310612 16318 raft_consensus.cc:348] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:45:47.310865 16318 raft_consensus.cc:374] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:45:47.310945 16318 raft_consensus.cc:729] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 411bce22959b49a88031e65d983d5244, State: Initialized, Role: FOLLOWER
I20260619 14:45:47.311547 16318 consensus_queue.cc:260] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:45:47.311411 16319 raft_consensus.cc:348] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:45:47.311688 16319 raft_consensus.cc:374] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:45:47.311782 16319 raft_consensus.cc:729] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e416f9f3d8104e42826ed1005c478b30, State: Initialized, Role: FOLLOWER
I20260619 14:45:47.312192 16166 heartbeater.cc:503] Master 127.0.98.62:45837 was elected leader, sending a full tablet report...
I20260619 14:45:47.312170 16317 raft_consensus.cc:348] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:45:47.312424 16317 raft_consensus.cc:374] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:45:47.312500 16317 raft_consensus.cc:729] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8504541fa3b84b0da8e0376ff83c6040, State: Initialized, Role: FOLLOWER
I20260619 14:45:47.312554 16318 ts_tablet_manager.cc:1428] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:45:47.312911 16317 consensus_queue.cc:260] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:45:47.313529 16319 consensus_queue.cc:260] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:45:47.313838 16317 ts_tablet_manager.cc:1428] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260619 14:45:47.314235 16032 heartbeater.cc:503] Master 127.0.98.62:45837 was elected leader, sending a full tablet report...
I20260619 14:45:47.314405 16319 ts_tablet_manager.cc:1428] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20260619 14:45:47.314571 16300 heartbeater.cc:503] Master 127.0.98.62:45837 was elected leader, sending a full tablet report...
I20260619 14:45:47.382790 16324 raft_consensus.cc:482] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:45:47.383080 16324 raft_consensus.cc:504] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:45:47.384419 16324 leader_election.cc:288] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 411bce22959b49a88031e65d983d5244 (127.0.98.2:39519), e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975)
I20260619 14:45:47.388414 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:47.384782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37841 (local address 127.0.98.2:39519)
0619 14:45:47.384919 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:47.384925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:47.385112 (+ 187us) server_negotiation.cc:407] Connection header received
0619 14:45:47.385214 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:47.385217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:47.385268 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:47.385369 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:47.385671 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.386272 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.387405 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.387854 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.387936 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:47.388037 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:47.388206 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 14:45:47.388285 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:45:47.388425 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:47.384638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39519 (local address 127.0.98.1:37841)
0619 14:45:47.385005 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:47.385020 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:47.385127 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:47.385383 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:47.385386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:47.385400 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:47.385552 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:47.385562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.386443 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.386445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:47.387282 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:47.387291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.387970 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.387977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:47.388075 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:47.388101 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:45:47.388178 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 14:45:47.388233 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":40,"thread_start_us":133,"threads_started":1}
I20260619 14:45:47.388525 16315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:47.384813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58393 (local address 127.0.98.3:35975)
0619 14:45:47.385002 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:47.385006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:47.385244 (+ 238us) server_negotiation.cc:407] Connection header received
0619 14:45:47.385356 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:47.385359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:47.385413 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:47.385493 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:45:47.385773 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.386247 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.387506 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.387941 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.388043 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:47.388104 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:47.388337 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 14:45:47.388403 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260619 14:45:47.388545 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:47.384655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35975 (local address 127.0.98.1:58393)
0619 14:45:47.385140 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:47.385153 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:45:47.385251 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:45:47.385526 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:45:47.385528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:45:47.385542 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:45:47.385661 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:47.385667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.386347 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.386354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:47.387316 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:45:47.387325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.388075 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.388079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:45:47.388210 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:47.388233 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:45:47.388309 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:45:47.388369 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":21,"thread_start_us":116,"threads_started":1}
I20260619 14:45:47.389132 16254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc6e2275f4c4785a81958f4a504b431" candidate_uuid: "8504541fa3b84b0da8e0376ff83c6040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e416f9f3d8104e42826ed1005c478b30" is_pre_election: true
I20260619 14:45:47.389372 16120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc6e2275f4c4785a81958f4a504b431" candidate_uuid: "8504541fa3b84b0da8e0376ff83c6040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "411bce22959b49a88031e65d983d5244" is_pre_election: true
I20260619 14:45:47.389468 16254 raft_consensus.cc:2454] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8504541fa3b84b0da8e0376ff83c6040 in term 0.
I20260619 14:45:47.389647 16120 raft_consensus.cc:2454] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8504541fa3b84b0da8e0376ff83c6040 in term 0.
I20260619 14:45:47.390065 15921 leader_election.cc:302] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8504541fa3b84b0da8e0376ff83c6040, e416f9f3d8104e42826ed1005c478b30; no voters:
I20260619 14:45:47.390324 16324 raft_consensus.cc:2781] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:45:47.390499 16324 raft_consensus.cc:482] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:45:47.390560 16324 raft_consensus.cc:3037] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:45:47.391712 16324 raft_consensus.cc:504] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:45:47.392095 16324 leader_election.cc:288] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 1 election: Requested vote from peers 411bce22959b49a88031e65d983d5244 (127.0.98.2:39519), e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975)
I20260619 14:45:47.392441 16120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc6e2275f4c4785a81958f4a504b431" candidate_uuid: "8504541fa3b84b0da8e0376ff83c6040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "411bce22959b49a88031e65d983d5244"
I20260619 14:45:47.392565 16120 raft_consensus.cc:3037] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:45:47.392602 16254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc6e2275f4c4785a81958f4a504b431" candidate_uuid: "8504541fa3b84b0da8e0376ff83c6040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e416f9f3d8104e42826ed1005c478b30"
I20260619 14:45:47.392760 16254 raft_consensus.cc:3037] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:45:47.393637 16120 raft_consensus.cc:2454] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8504541fa3b84b0da8e0376ff83c6040 in term 1.
I20260619 14:45:47.393899 16254 raft_consensus.cc:2454] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8504541fa3b84b0da8e0376ff83c6040 in term 1.
I20260619 14:45:47.393918 15919 leader_election.cc:302] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411bce22959b49a88031e65d983d5244, 8504541fa3b84b0da8e0376ff83c6040; no voters:
I20260619 14:45:47.394094 16324 raft_consensus.cc:2781] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:45:47.394348 16324 raft_consensus.cc:686] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 LEADER]: Becoming Leader. State: Replica: 8504541fa3b84b0da8e0376ff83c6040, State: Running, Role: LEADER
I20260619 14:45:47.394727 16324 consensus_queue.cc:237] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:45:47.398176 15841 catalog_manager.cc:5495] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8504541fa3b84b0da8e0376ff83c6040 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8504541fa3b84b0da8e0376ff83c6040" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } health_report { overall_health: HEALTHY } } }
I20260619 14:45:47.415989 16316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:47.412402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59692 (local address 127.0.98.1:45829)
0619 14:45:47.412541 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:45:47.412546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:45:47.412592 (+ 46us) server_negotiation.cc:407] Connection header received
0619 14:45:47.412760 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:45:47.412762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:45:47.412816 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:45:47.412901 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:45:47.413140 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.413621 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.414524 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:45:47.414721 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:45:47.414793 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:45:47.415118 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:45:47.415603 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:45:47.415694 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:45:47.415779 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 14:45:47.415824 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:45:47.421638 16120 raft_consensus.cc:1261] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244 [term 1 FOLLOWER]: Refusing update from remote peer 8504541fa3b84b0da8e0376ff83c6040: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:45:47.421677 16254 raft_consensus.cc:1261] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30 [term 1 FOLLOWER]: Refusing update from remote peer 8504541fa3b84b0da8e0376ff83c6040: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:45:47.422377 16328 consensus_queue.cc:1035] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:45:47.422652 16324 consensus_queue.cc:1035] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:45:47.431138 16332 mvcc.cc:205] Tried to move back new op lower bound from 7298581903032094720 to 7298581902931980288. Current Snapshot: MvccSnapshot[applied={T|T < 7298581903032094720}]
I20260619 14:45:47.439656 15839 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:45:47.439807 15839 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:45:47.442225 15839 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: Sending DeleteTablet for 3 replicas of tablet dcc6e2275f4c4785a81958f4a504b431
I20260619 14:45:47.442994 16234 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc6e2275f4c4785a81958f4a504b431 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:45:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41778
I20260619 14:45:47.443064 15966 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc6e2275f4c4785a81958f4a504b431 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:45:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59676
I20260619 14:45:47.443043 16100 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc6e2275f4c4785a81958f4a504b431 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:45:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58744
I20260619 14:45:47.443645 16341 tablet_replica.cc:330] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30: stopping tablet replica
I20260619 14:45:47.443666 16342 tablet_replica.cc:330] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244: stopping tablet replica
I20260619 14:45:47.443692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 15903
I20260619 14:45:47.444778 16342 raft_consensus.cc:2229] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:45:47.445094 16342 raft_consensus.cc:2258] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:45:47.446379 16342 ts_tablet_manager.cc:1905] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:45:47.448685 16342 ts_tablet_manager.cc:1918] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:45:47.448792 16342 log.cc:1196] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/dcc6e2275f4c4785a81958f4a504b431
I20260619 14:45:47.449162 16342 ts_tablet_manager.cc:1939] T dcc6e2275f4c4785a81958f4a504b431 P 411bce22959b49a88031e65d983d5244: Deleting consensus metadata
I20260619 14:45:47.449134 16341 raft_consensus.cc:2229] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:45:47.449471 16341 raft_consensus.cc:2258] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:45:47.450551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 16037
W20260619 14:45:47.451399 15826 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45829 (error 108)
I20260619 14:45:47.452553 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:45:47.452285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45829 (local address 127.0.0.1:59696)
0619 14:45:47.452394 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 14:45:47.452459 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 14:45:47.452536 16341 ts_tablet_manager.cc:1905] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:45:47.452965 15825 catalog_manager.cc:4841] TS 411bce22959b49a88031e65d983d5244 (127.0.98.2:39519): tablet dcc6e2275f4c4785a81958f4a504b431 (table test-table [id=ea5aa4bb2cb54c7ca82618dee1aaf07b]) successfully deleted
W20260619 14:45:47.453220 15826 catalog_manager.cc:4584] TS 8504541fa3b84b0da8e0376ff83c6040 (127.0.98.1:45829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dcc6e2275f4c4785a81958f4a504b431: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45829: connect: Connection refused (error 111)
I20260619 14:45:47.456288 16341 ts_tablet_manager.cc:1918] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:45:47.456422 16341 log.cc:1196] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/dcc6e2275f4c4785a81958f4a504b431
I20260619 14:45:47.456799 16341 ts_tablet_manager.cc:1939] T dcc6e2275f4c4785a81958f4a504b431 P e416f9f3d8104e42826ed1005c478b30: Deleting consensus metadata
I20260619 14:45:47.458233 15827 catalog_manager.cc:4841] TS e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975): tablet dcc6e2275f4c4785a81958f4a504b431 (table test-table [id=ea5aa4bb2cb54c7ca82618dee1aaf07b]) successfully deleted
I20260619 14:45:47.458942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 16171
I20260619 14:45:47.464650 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 15809
I20260619 14:46:07.472083 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45837
--webserver_interface=127.0.98.62
--webserver_port=46491
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:07.584916 16344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:07.585201 16344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:07.585243 16344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:07.589421 16344 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 14:46:07.589481 16344 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 14:46:07.589498 16344 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 14:46:07.589511 16344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:07.589530 16344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:07.589548 16344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:07.589587 16344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:07.589620 16344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:07.595081 16344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:07.596205 16344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:07.597245 16344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:07.610754 16351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:07.610720 16353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:07.610698 16350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:07.610715 16344 server_base.cc:1034] running on GCE node
I20260619 14:46:07.611730 16344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:07.612874 16344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:07.614037 16344 hybrid_clock.cc:648] HybridClock initialized: now 1781880367614021 us; error 29 us; skew 500 ppm
Jun 19 14:46:07 dist-test-slave-cr22 krb5kdc[15796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:07.617188 16344 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:46:07.618167 16344 webserver.cc:460] Webserver started at http://127.0.98.62:46491/ using document root <none> and password file <none>
I20260619 14:46:07.618779 16344 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:07.618894 16344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:07.623001 16344 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:46:07.625285 16360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:07.626572 16344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:46:07.626740 16344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "1fea269cd46749f88370613b756334a2"
format_stamp: "Formatted at 2026-06-19 14:45:46 on dist-test-slave-cr22"
server_key: "36403746de8bf71fb0570e6de8c709ec"
server_key_iv: "db02aae0cd6eec9b2ac4cd9d391b0227"
server_key_version: "encryptionkey@0"
I20260619 14:46:07.627385 16344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:07.638830 16344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:07.641856 16344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:07.642066 16344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:07.649353 16344 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45837
I20260619 14:46:07.649657 16412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45837 every 8 connection(s)
I20260619 14:46:07.650482 16344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:07.655079 16413 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:46:07.656853 16413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: Bootstrap starting.
I20260619 14:46:07.659101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 16344
I20260619 14:46:07.659631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45829
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45837
--webserver_port=37193
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:46:07.674813 16413 log.cc:824] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:07.683785 16413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:46:07.684176 16413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: Bootstrap complete.
I20260619 14:46:07.688275 16413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fea269cd46749f88370613b756334a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45837 } }
I20260619 14:46:07.688728 16413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fea269cd46749f88370613b756334a2, State: Initialized, Role: FOLLOWER
I20260619 14:46:07.689260 16413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fea269cd46749f88370613b756334a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45837 } }
I20260619 14:46:07.689378 16413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:07.689476 16413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:07.689590 16413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:46:07.691573 16413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fea269cd46749f88370613b756334a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45837 } }
I20260619 14:46:07.691893 16413 leader_election.cc:302] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fea269cd46749f88370613b756334a2; no voters:
I20260619 14:46:07.692203 16413 leader_election.cc:288] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:46:07.692286 16417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:46:07.692564 16417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 2 LEADER]: Becoming Leader. State: Replica: 1fea269cd46749f88370613b756334a2, State: Running, Role: LEADER
I20260619 14:46:07.692951 16417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fea269cd46749f88370613b756334a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45837 } }
I20260619 14:46:07.693580 16413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:07.694012 16417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fea269cd46749f88370613b756334a2. Latest consensus state: current_term: 2 leader_uuid: "1fea269cd46749f88370613b756334a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fea269cd46749f88370613b756334a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45837 } } }
I20260619 14:46:07.694157 16417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:07.694489 16418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1fea269cd46749f88370613b756334a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fea269cd46749f88370613b756334a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45837 } } }
I20260619 14:46:07.694674 16418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:07.694566 16421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:07.699625 16421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:07.700346 16421 catalog_manager.cc:1259] Loaded cluster ID: e5217a81c1d341539d09a86632200d86
I20260619 14:46:07.700419 16421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:07.702812 16421 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:07.703617 16421 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: Loaded TSK: 0
I20260619 14:46:07.722084 16421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: Generated new TSK 1
I20260619 14:46:07.722483 16421 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:46:07.780033 16415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:07.780318 16415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:07.780364 16415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:07.784497 16415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:07.784562 16415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:07.784654 16415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:07.789983 16415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37193
--tserver_master_addrs=127.0.98.62:45837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:07.790942 16415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:07.791962 16415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:07.801530 16441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:07.801548 16440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:07.801548 16443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:07.802189 16415 server_base.cc:1034] running on GCE node
I20260619 14:46:07.802646 16415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:07.803304 16415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:07.804497 16415 hybrid_clock.cc:648] HybridClock initialized: now 1781880367804470 us; error 44 us; skew 500 ppm
Jun 19 14:46:07 dist-test-slave-cr22 krb5kdc[15796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:07.807564 16415 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:46:07.808687 16415 webserver.cc:460] Webserver started at http://127.0.98.1:37193/ using document root <none> and password file <none>
I20260619 14:46:07.809393 16415 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:07.809514 16415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:07.814610 16415 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:46:07.817334 16450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:07.818902 16415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:46:07.819046 16415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "8504541fa3b84b0da8e0376ff83c6040"
format_stamp: "Formatted at 2026-06-19 14:45:46 on dist-test-slave-cr22"
server_key: "b7b114f20f265bbc27b5dacd35bde55e"
server_key_iv: "0db6786587a3a9b4dd80179df0cd0394"
server_key_version: "encryptionkey@0"
I20260619 14:46:07.819733 16415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:07.841240 16415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:07.844815 16415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:07.845026 16415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:07.845634 16415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:07.846729 16457 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:46:07.850966 16415 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:46:07.851070 16415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 14:46:07.851155 16415 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:46:07.853056 16415 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:46:07.853132 16415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 14:46:07.853515 16457 tablet_bootstrap.cc:492] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040: Bootstrap starting.
I20260619 14:46:07.864493 16415 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45829
I20260619 14:46:07.864594 16564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45829 every 8 connection(s)
I20260619 14:46:07.865583 16415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:07.866585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 16415
I20260619 14:46:07.867096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39519
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45837
--webserver_port=41755
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:46:07.876559 16457 log.cc:824] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:46:07 dist-test-slave-cr22 krb5kdc[15796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:07.886258 16457 tablet_bootstrap.cc:492] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:46:07.886678 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:07.868119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:40327)
0619 14:46:07.868685 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:07.868742 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:07.870114 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:07.871392 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:07.871401 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:07.871883 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:07.872328 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:07.872342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:07.874362 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:07.874369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:07.876088 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:07.876103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:07.877217 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:07.877221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:07.877438 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:07.878234 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:07.878256 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:07.880408 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:07.883202 (+ 2794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:07.883209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:07.883223 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:07.883613 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:07.884042 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:07.884045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:07.884047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:07.884208 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:07.884932 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:07.884938 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:07.885223 (+ 285us) client_negotiation.cc:769] Sending connection context
0619 14:46:07.885409 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 14:46:07.885737 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":154,"threads_started":1}
I20260619 14:46:07.886925 16457 tablet_bootstrap.cc:492] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040: Bootstrap complete.
I20260619 14:46:07.888265 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:07.868092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40327 (local address 127.0.98.62:45837)
0619 14:46:07.868572 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:07.868596 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:07.868888 (+ 292us) server_negotiation.cc:407] Connection header received
0619 14:46:07.870641 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:07.870651 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:07.870976 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:07.871405 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:07.872507 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:07.874124 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:07.876238 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:07.877042 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:07.877264 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:07.880643 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:07.880650 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:07.880667 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:07.880701 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:07.883042 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:07.883744 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:07.883750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:07.883756 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:07.883904 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:07.884324 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:07.884327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:07.884329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:07.884756 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:07.884878 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:07.887113 (+ 2235us) server_negotiation.cc:299] Negotiation successful
0619 14:46:07.887420 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":135,"threads_started":1}
I20260619 14:46:07.889343 16457 ts_tablet_manager.cc:1397] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040: Time spent bootstrapping tablet: real 0.036s user 0.016s sys 0.016s
I20260619 14:46:07.890688 16565 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45837
I20260619 14:46:07.890923 16565 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:07.891482 16565 heartbeater.cc:511] Master 127.0.98.62:45837 requested a full tablet report, sending...
I20260619 14:46:07.892428 16457 raft_consensus.cc:348] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:46:07.892966 16457 raft_consensus.cc:729] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8504541fa3b84b0da8e0376ff83c6040, State: Initialized, Role: FOLLOWER
I20260619 14:46:07.893579 16457 consensus_queue.cc:260] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:46:07.894304 16376 ts_manager.cc:194] Registered new tserver with Master: 8504541fa3b84b0da8e0376ff83c6040 (127.0.98.1:45829)
I20260619 14:46:07.894785 16457 ts_tablet_manager.cc:1428] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260619 14:46:07.898399 16376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40327
I20260619 14:46:07.901242 16565 heartbeater.cc:503] Master 127.0.98.62:45837 was elected leader, sending a full tablet report...
Jun 19 14:46:07 dist-test-slave-cr22 krb5kdc[15796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:46:07.909231 16572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:07.897810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45829 (local address 127.0.0.1:37652)
0619 14:46:07.898585 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:07.898637 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:07.898819 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:07.899465 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:07.899471 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:07.899884 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:07.899995 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:07.900002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:07.901045 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:07.901052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:07.901943 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:07.901954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:07.902690 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:07.902713 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:07.902838 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:07.903585 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:07.903597 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:07.905189 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:07.907373 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:07.907381 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:07.907385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:07.907699 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:07.908087 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:07.908090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:07.908091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:07.908177 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:07.908701 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:07.908706 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:07.908821 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 14:46:07.908941 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 14:46:07.909101 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":138,"threads_started":1}
I20260619 14:46:07.909438 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:07.897866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37652 (local address 127.0.98.1:45829)
0619 14:46:07.898297 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:07.898309 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:07.898750 (+ 441us) server_negotiation.cc:407] Connection header received
0619 14:46:07.899043 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:07.899049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:07.899277 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:07.899486 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:07.900120 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:07.900923 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:07.902086 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:07.902573 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:07.902688 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:07.905342 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:07.905352 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:07.905365 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:07.905392 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:07.907243 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:07.907861 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:07.907867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:07.907869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:07.907962 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:07.908280 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:07.908283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:07.908285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:07.908587 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:07.908715 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:07.909074 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 14:46:07.909252 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":318,"thread_start_us":96,"threads_started":1}
I20260619 14:46:07.910501 16499 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc6e2275f4c4785a81958f4a504b431 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:45:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37652
I20260619 14:46:07.911367 16574 tablet_replica.cc:330] stopping tablet replica
I20260619 14:46:07.911721 16574 raft_consensus.cc:2229] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:07.911981 16574 raft_consensus.cc:2258] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:07.913196 16574 ts_tablet_manager.cc:1905] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:07.917804 16574 ts_tablet_manager.cc:1918] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:07.917907 16574 log.cc:1196] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/wal/wals/dcc6e2275f4c4785a81958f4a504b431
I20260619 14:46:07.918182 16574 ts_tablet_manager.cc:1939] T dcc6e2275f4c4785a81958f4a504b431 P 8504541fa3b84b0da8e0376ff83c6040: Deleting consensus metadata
I20260619 14:46:07.920260 16362 catalog_manager.cc:4841] TS 8504541fa3b84b0da8e0376ff83c6040 (127.0.98.1:45829): tablet dcc6e2275f4c4785a81958f4a504b431 (table test-table [id=ea5aa4bb2cb54c7ca82618dee1aaf07b]) successfully deleted
W20260619 14:46:07.995703 16569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:07.995970 16569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:07.996052 16569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:08.000648 16569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:08.000715 16569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:08.000813 16569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:08.006271 16569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41755
--tserver_master_addrs=127.0.98.62:45837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:08.007404 16569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:08.008481 16569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:08.018477 16579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:08.018467 16580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:08.018971 16569 server_base.cc:1034] running on GCE node
W20260619 14:46:08.018581 16582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:08.020331 16569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:08.021066 16569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:08.022270 16569 hybrid_clock.cc:648] HybridClock initialized: now 1781880368022128 us; error 157 us; skew 500 ppm
Jun 19 14:46:08 dist-test-slave-cr22 krb5kdc[15796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:08.025588 16569 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:46:08.026592 16569 webserver.cc:460] Webserver started at http://127.0.98.2:41755/ using document root <none> and password file <none>
I20260619 14:46:08.027185 16569 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:08.027302 16569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:08.031533 16569 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:46:08.033797 16589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:08.035174 16569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:46:08.035327 16569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "411bce22959b49a88031e65d983d5244"
format_stamp: "Formatted at 2026-06-19 14:45:46 on dist-test-slave-cr22"
server_key: "9db852459ecdfdf8da62abe158911c92"
server_key_iv: "159bcfcd24e32b3d804007c0c93f63bb"
server_key_version: "encryptionkey@0"
I20260619 14:46:08.035938 16569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:08.048555 16569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:08.052222 16569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:08.052450 16569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:08.053061 16569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:08.053930 16569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:08.054025 16569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:08.054116 16569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:08.054162 16569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:08.064039 16569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39519
I20260619 14:46:08.064181 16702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39519 every 8 connection(s)
I20260619 14:46:08.065107 16569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:08.067615 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 16569
I20260619 14:46:08.068145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35975
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45837
--webserver_port=41801
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:46:08 dist-test-slave-cr22 krb5kdc[15796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:08.081346 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:08.066962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42129 (local address 127.0.98.62:45837)
0619 14:46:08.067106 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:08.067109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:08.068336 (+ 1227us) server_negotiation.cc:407] Connection header received
0619 14:46:08.069791 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:08.069796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:08.069873 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:08.069990 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:08.071424 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.072055 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.073975 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.074221 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.074396 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:08.077548 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:08.077551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:08.077554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:08.077584 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:08.079105 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:08.079721 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:08.079725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:08.079731 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:08.079811 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:08.080169 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:08.080173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:08.080174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:08.080369 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:08.080478 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:08.081018 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 14:46:08.081162 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:46:08.082158 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:08.067307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:42129)
0619 14:46:08.068164 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:08.068222 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:08.069517 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:08.070142 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:08.070148 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:08.070540 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:08.071163 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:08.071185 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.072197 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.072204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:08.073821 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:08.073835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.074399 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.074406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:08.074665 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:08.075402 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:08.075428 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:08.077337 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:08.079237 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:08.079243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:08.079262 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:08.079616 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:08.079910 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:08.079914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:08.079916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:08.080067 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:08.080488 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:08.080497 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:08.080767 (+ 270us) client_negotiation.cc:769] Sending connection context
0619 14:46:08.080955 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 14:46:08.081298 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":164,"threads_started":1}
I20260619 14:46:08.083348 16703 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45837
I20260619 14:46:08.083639 16703 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:08.084167 16703 heartbeater.cc:511] Master 127.0.98.62:45837 requested a full tablet report, sending...
I20260619 14:46:08.085188 16376 ts_manager.cc:194] Registered new tserver with Master: 411bce22959b49a88031e65d983d5244 (127.0.98.2:39519)
I20260619 14:46:08.085765 16376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42129
W20260619 14:46:08.181720 16706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:08.182036 16706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:08.182111 16706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:08.186257 16706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:08.186348 16706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:08.186524 16706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:08.192255 16706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41801
--tserver_master_addrs=127.0.98.62:45837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:08.193473 16706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:08.194680 16706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:08.204795 16714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:08.204844 16712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:08.204855 16711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:08.205209 16706 server_base.cc:1034] running on GCE node
I20260619 14:46:08.205557 16706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:08.206123 16706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:08.207309 16706 hybrid_clock.cc:648] HybridClock initialized: now 1781880368207262 us; error 53 us; skew 500 ppm
Jun 19 14:46:08 dist-test-slave-cr22 krb5kdc[15796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:08.210347 16706 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:46:08.211268 16706 webserver.cc:460] Webserver started at http://127.0.98.3:41801/ using document root <none> and password file <none>
I20260619 14:46:08.211905 16706 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:08.212014 16706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:08.216131 16706 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:46:08.218480 16721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:08.219950 16706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:08.220095 16706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "e416f9f3d8104e42826ed1005c478b30"
format_stamp: "Formatted at 2026-06-19 14:45:47 on dist-test-slave-cr22"
server_key: "7fa2191f19e9ad0a012a4e6ba7085737"
server_key_iv: "c1933ecfc8e74392b51c5ab5fb9010e6"
server_key_version: "encryptionkey@0"
I20260619 14:46:08.220754 16706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:08.237152 16706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:08.240638 16706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:08.240912 16706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:08.241514 16706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:08.242357 16706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:08.242440 16706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:08.242511 16706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:08.242555 16706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:08.253017 16706 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35975
I20260619 14:46:08.253094 16834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35975 every 8 connection(s)
I20260619 14:46:08.254199 16706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:08.255609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 16706
Jun 19 14:46:08 dist-test-slave-cr22 krb5kdc[15796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:08.267385 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:08.256757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.0.98.62:45837)
0619 14:46:08.256953 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:08.256959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:08.257622 (+ 663us) server_negotiation.cc:407] Connection header received
0619 14:46:08.257830 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:08.257834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:08.257887 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:08.257945 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:08.259184 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.260179 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.261559 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.261878 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.261987 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:08.263796 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:08.263799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:08.263801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:08.263825 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:08.265549 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:08.265998 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:08.266002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:08.266004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:08.266085 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:08.266413 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:08.266419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:08.266424 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:08.266618 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:08.266764 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:08.267030 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 14:46:08.267181 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 14:46:08.271857 16838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:08.256858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54781 (local address 127.0.98.62:45837)
0619 14:46:08.257070 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:08.257075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:08.257992 (+ 917us) server_negotiation.cc:407] Connection header received
0619 14:46:08.259302 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:08.259306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:08.259355 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:08.259574 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:08.261421 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.262228 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.264402 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.264787 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.264950 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:08.267740 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:08.267744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:08.267746 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:08.267770 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:08.269774 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:08.270287 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:08.270290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:08.270291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:08.270372 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:08.270768 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:08.270771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:08.270772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:08.270949 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:08.271035 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:08.271610 (+ 575us) server_negotiation.cc:299] Negotiation successful
0619 14:46:08.271723 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":61,"threads_started":1}
I20260619 14:46:08.272794 16839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:08.256987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.3:54781)
0619 14:46:08.257844 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:08.257903 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:08.259063 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:08.259907 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:08.259916 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:08.260528 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:08.261212 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:08.261233 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.262707 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.262711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:08.264281 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:08.264292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.265018 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.265022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:08.265173 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:08.265749 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:08.265765 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:08.267569 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:08.269907 (+ 2338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:08.269912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:08.269924 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:08.270195 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:08.270506 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:08.270508 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:08.270510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:08.270657 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:08.271045 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:08.271050 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:08.271291 (+ 241us) client_negotiation.cc:769] Sending connection context
0619 14:46:08.271553 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 14:46:08.271827 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":143,"threads_started":1}
I20260619 14:46:08.274047 16835 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45837
I20260619 14:46:08.274305 16835 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:08.274827 16835 heartbeater.cc:511] Master 127.0.98.62:45837 requested a full tablet report, sending...
I20260619 14:46:08.276114 16376 ts_manager.cc:194] Registered new tserver with Master: e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975)
I20260619 14:46:08.276697 16376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54781
I20260619 14:46:08.281730 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:46:08.286381 16838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:08.282334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34164 (local address 127.0.98.62:45837)
0619 14:46:08.282538 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:08.282543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:08.282717 (+ 174us) server_negotiation.cc:407] Connection header received
0619 14:46:08.282867 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:08.282869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:08.282979 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:08.283080 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:08.283381 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.284093 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.285069 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.285320 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.285409 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:08.285892 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:08.286043 (+ 151us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 14:46:08.286267 (+ 224us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34164: token expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 14:46:08.286556 16838 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34164: token expired
W20260619 14:46:08.286872 16304 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:45837)
I20260619 14:46:08.295344 16838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:08.287259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.0.98.62:45837)
0619 14:46:08.287378 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:08.287381 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:08.287393 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:46:08.287536 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:08.287538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:08.287584 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:08.287653 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:08.288336 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.288893 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.289746 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.290017 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.290103 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:08.291626 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:08.291629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:08.291632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:08.291658 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:08.293391 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:08.293836 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:08.293839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:08.293841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:08.293924 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:08.294283 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:08.294286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:08.294288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:08.294536 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:08.294759 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:08.295010 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 14:46:08.295111 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:46:08.298027 16376 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:46:08.301481 16376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:46:08.310087 16499 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea055d409744ea695a6ddec2687ebc7 (DEFAULT_TABLE table=test-table [id=07500dab3fd54f3b8f9ce0bee0918a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:08.310812 16499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ea055d409744ea695a6ddec2687ebc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:08.314069 16457 tablet_bootstrap.cc:492] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040: Bootstrap starting.
I20260619 14:46:08.314512 16572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:08.308970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35975 (local address 127.0.0.1:55954)
0619 14:46:08.309152 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:08.309168 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:08.309254 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:08.310524 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:08.310527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:08.310557 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:08.310670 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:08.310678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.311823 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.311827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:08.313044 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:08.313055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.314050 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.314053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:08.314168 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:08.314222 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 14:46:08.314278 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 14:46:08.314362 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":96,"spinlock_wait_cycles":3456}
I20260619 14:46:08.315021 16457 tablet_bootstrap.cc:654] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:08.315047 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:08.309166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39519 (local address 127.0.0.1:42260)
0619 14:46:08.309514 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:08.309529 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:08.309617 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:08.310530 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:08.310533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:08.310566 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:08.310674 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:08.310682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.312335 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.312338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:08.313455 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:08.313464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.314610 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.314616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:08.314723 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:08.314747 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:46:08.314814 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 14:46:08.314887 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":3,"spinlock_wait_cycles":3328,"thread_start_us":148,"threads_started":1}
I20260619 14:46:08.315214 16845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:08.309505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.0.98.2:39519)
0619 14:46:08.309971 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:08.309984 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:08.310011 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:46:08.310095 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:08.310101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:08.310328 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:08.310840 (+ 512us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:08.311022 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.312210 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.313583 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.314465 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.314728 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:08.314898 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:08.314953 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 14:46:08.315052 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":120,"threads_started":1}
I20260619 14:46:08.315753 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:08.309687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55954 (local address 127.0.98.3:35975)
0619 14:46:08.310020 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:08.310036 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:08.310055 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:46:08.310118 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:08.310122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:08.310313 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:08.310541 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:08.310768 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.311712 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.313184 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.313893 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.314927 (+ 1034us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:08.315386 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:08.315490 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 14:46:08.315601 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":145,"threads_started":1}
I20260619 14:46:08.317351 16457 tablet_bootstrap.cc:492] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040: No bootstrap required, opened a new log
I20260619 14:46:08.317479 16457 ts_tablet_manager.cc:1397] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:46:08.317907 16457 raft_consensus.cc:348] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:46:08.318045 16457 raft_consensus.cc:374] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:08.318097 16457 raft_consensus.cc:729] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8504541fa3b84b0da8e0376ff83c6040, State: Initialized, Role: FOLLOWER
I20260619 14:46:08.318264 16457 consensus_queue.cc:260] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:46:08.318558 16457 ts_tablet_manager.cc:1428] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:46:08.318512 16637 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea055d409744ea695a6ddec2687ebc7 (DEFAULT_TABLE table=test-table [id=07500dab3fd54f3b8f9ce0bee0918a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:08.318509 16769 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea055d409744ea695a6ddec2687ebc7 (DEFAULT_TABLE table=test-table [id=07500dab3fd54f3b8f9ce0bee0918a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:08.319641 16769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ea055d409744ea695a6ddec2687ebc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:08.319636 16637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ea055d409744ea695a6ddec2687ebc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:08.327899 16849 tablet_bootstrap.cc:492] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30: Bootstrap starting.
I20260619 14:46:08.328534 16848 tablet_bootstrap.cc:492] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244: Bootstrap starting.
I20260619 14:46:08.330848 16849 tablet_bootstrap.cc:654] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:08.331342 16848 tablet_bootstrap.cc:654] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:08.332324 16849 log.cc:824] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:08.332324 16848 log.cc:824] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:08.334259 16848 tablet_bootstrap.cc:492] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244: No bootstrap required, opened a new log
I20260619 14:46:08.334281 16849 tablet_bootstrap.cc:492] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30: No bootstrap required, opened a new log
I20260619 14:46:08.334486 16849 ts_tablet_manager.cc:1397] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260619 14:46:08.334656 16848 ts_tablet_manager.cc:1397] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:46:08.337637 16848 raft_consensus.cc:348] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:46:08.337874 16848 raft_consensus.cc:374] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:08.337684 16849 raft_consensus.cc:348] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:46:08.337940 16848 raft_consensus.cc:729] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 411bce22959b49a88031e65d983d5244, State: Initialized, Role: FOLLOWER
I20260619 14:46:08.337934 16849 raft_consensus.cc:374] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:08.338013 16849 raft_consensus.cc:729] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e416f9f3d8104e42826ed1005c478b30, State: Initialized, Role: FOLLOWER
I20260619 14:46:08.338418 16848 consensus_queue.cc:260] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:46:08.338444 16849 consensus_queue.cc:260] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:46:08.339114 16835 heartbeater.cc:503] Master 127.0.98.62:45837 was elected leader, sending a full tablet report...
I20260619 14:46:08.339115 16703 heartbeater.cc:503] Master 127.0.98.62:45837 was elected leader, sending a full tablet report...
I20260619 14:46:08.339538 16848 ts_tablet_manager.cc:1428] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:46:08.339572 16849 ts_tablet_manager.cc:1428] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30: Time spent starting tablet: real 0.005s user 0.005s sys 0.002s
W20260619 14:46:08.367221 16566 tablet.cc:2369] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:46:08.411623 16852 raft_consensus.cc:482] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:08.411810 16852 raft_consensus.cc:504] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:46:08.412963 16852 leader_election.cc:288] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975), 8504541fa3b84b0da8e0376ff83c6040 (127.0.98.1:45829)
I20260619 14:46:08.418892 16855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:08.413431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48339 (local address 127.0.98.1:45829)
0619 14:46:08.413697 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:08.413702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:08.413716 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:46:08.413769 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:08.413775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:08.413846 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:08.413963 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:08.414262 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.415107 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.417223 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.417931 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.418028 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:08.418190 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:08.418652 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 14:46:08.418731 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
I20260619 14:46:08.418879 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:08.413270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56871 (local address 127.0.98.3:35975)
0619 14:46:08.413583 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:08.413588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:08.413605 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:46:08.413660 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:08.413663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:08.413718 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:08.413807 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:08.414176 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.415017 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.417223 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.417889 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.418017 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:08.418083 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:08.418652 (+ 569us) server_negotiation.cc:299] Negotiation successful
0619 14:46:08.418726 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260619 14:46:08.418958 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:08.413229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45829 (local address 127.0.98.2:48339)
0619 14:46:08.413561 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:08.413576 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:08.413663 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:08.413993 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:08.413998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:08.414015 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:08.414134 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:08.414143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.415205 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.415208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:08.417058 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:08.417070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.418064 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.418068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:08.418402 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:08.418531 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 14:46:08.418619 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 14:46:08.418770 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":387,"thread_start_us":116,"threads_started":1}
I20260619 14:46:08.419158 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:08.413139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35975 (local address 127.0.98.2:56871)
0619 14:46:08.413269 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:08.413287 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:08.413467 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:08.413852 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:08.413854 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:08.413872 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:08.414032 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:08.414040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.415126 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.415129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:08.417058 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:08.417070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.418064 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.418068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:08.418407 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:08.418527 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 14:46:08.418619 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 14:46:08.418771 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":407}
I20260619 14:46:08.419684 16519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea055d409744ea695a6ddec2687ebc7" candidate_uuid: "411bce22959b49a88031e65d983d5244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8504541fa3b84b0da8e0376ff83c6040" is_pre_election: true
I20260619 14:46:08.419888 16789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea055d409744ea695a6ddec2687ebc7" candidate_uuid: "411bce22959b49a88031e65d983d5244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e416f9f3d8104e42826ed1005c478b30" is_pre_election: true
I20260619 14:46:08.420034 16519 raft_consensus.cc:2454] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411bce22959b49a88031e65d983d5244 in term 0.
I20260619 14:46:08.420213 16789 raft_consensus.cc:2454] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411bce22959b49a88031e65d983d5244 in term 0.
I20260619 14:46:08.420473 16591 leader_election.cc:302] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411bce22959b49a88031e65d983d5244, 8504541fa3b84b0da8e0376ff83c6040; no voters:
I20260619 14:46:08.420753 16852 raft_consensus.cc:2781] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:46:08.420840 16852 raft_consensus.cc:482] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:46:08.420884 16852 raft_consensus.cc:3037] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:08.421896 16852 raft_consensus.cc:504] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:46:08.422227 16852 leader_election.cc:288] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [CANDIDATE]: Term 1 election: Requested vote from peers e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975), 8504541fa3b84b0da8e0376ff83c6040 (127.0.98.1:45829)
I20260619 14:46:08.422652 16789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea055d409744ea695a6ddec2687ebc7" candidate_uuid: "411bce22959b49a88031e65d983d5244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e416f9f3d8104e42826ed1005c478b30"
I20260619 14:46:08.422636 16519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea055d409744ea695a6ddec2687ebc7" candidate_uuid: "411bce22959b49a88031e65d983d5244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8504541fa3b84b0da8e0376ff83c6040"
I20260619 14:46:08.422755 16789 raft_consensus.cc:3037] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:08.422782 16519 raft_consensus.cc:3037] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:08.423904 16789 raft_consensus.cc:2454] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411bce22959b49a88031e65d983d5244 in term 1.
I20260619 14:46:08.424060 16519 raft_consensus.cc:2454] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411bce22959b49a88031e65d983d5244 in term 1.
I20260619 14:46:08.424230 16592 leader_election.cc:302] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411bce22959b49a88031e65d983d5244, e416f9f3d8104e42826ed1005c478b30; no voters:
I20260619 14:46:08.424391 16852 raft_consensus.cc:2781] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:08.424597 16852 raft_consensus.cc:686] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [term 1 LEADER]: Becoming Leader. State: Replica: 411bce22959b49a88031e65d983d5244, State: Running, Role: LEADER
I20260619 14:46:08.424908 16852 consensus_queue.cc:237] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:46:08.428402 16376 catalog_manager.cc:5495] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 reported cstate change: term changed from 0 to 1, leader changed from <none> to 411bce22959b49a88031e65d983d5244 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "411bce22959b49a88031e65d983d5244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20260619 14:46:08.442036 16845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:08.438005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42268 (local address 127.0.98.2:39519)
0619 14:46:08.438138 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:08.438142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:08.438157 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:46:08.438294 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:08.438296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:08.438388 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:08.438481 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:08.438752 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.439259 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.440190 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:08.440503 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:08.440603 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:08.440920 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:08.441426 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:08.441528 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:08.441712 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 14:46:08.441826 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:46:08.448841 16519 raft_consensus.cc:1261] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Refusing update from remote peer 411bce22959b49a88031e65d983d5244: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:08.448843 16789 raft_consensus.cc:1261] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30 [term 1 FOLLOWER]: Refusing update from remote peer 411bce22959b49a88031e65d983d5244: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:08.449795 16852 consensus_queue.cc:1035] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:08.450057 16856 consensus_queue.cc:1035] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:08.467095 16377 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:08.467299 16377 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:08.470198 16377 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: Sending DeleteTablet for 3 replicas of tablet 7ea055d409744ea695a6ddec2687ebc7
I20260619 14:46:08.471032 16769 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ea055d409744ea695a6ddec2687ebc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55954
I20260619 14:46:08.471095 16637 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ea055d409744ea695a6ddec2687ebc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42260
I20260619 14:46:08.471640 16499 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ea055d409744ea695a6ddec2687ebc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37652
I20260619 14:46:08.471688 16869 tablet_replica.cc:330] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244: stopping tablet replica
I20260619 14:46:08.472074 16869 raft_consensus.cc:2229] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:46:08.472366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 16415
I20260619 14:46:08.472476 16869 raft_consensus.cc:2258] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:08.475068 16869 ts_tablet_manager.cc:1905] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:08.479203 16869 ts_tablet_manager.cc:1918] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:08.479351 16869 log.cc:1196] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/7ea055d409744ea695a6ddec2687ebc7
I20260619 14:46:08.479765 16869 ts_tablet_manager.cc:1939] T 7ea055d409744ea695a6ddec2687ebc7 P 411bce22959b49a88031e65d983d5244: Deleting consensus metadata
I20260619 14:46:08.480188 16870 tablet_replica.cc:330] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30: stopping tablet replica
I20260619 14:46:08.480659 16870 raft_consensus.cc:2229] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:08.481076 16870 raft_consensus.cc:2258] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:08.481206 16361 catalog_manager.cc:4841] TS 411bce22959b49a88031e65d983d5244 (127.0.98.2:39519): tablet 7ea055d409744ea695a6ddec2687ebc7 (table test-table [id=07500dab3fd54f3b8f9ce0bee0918a0d]) successfully deleted
W20260619 14:46:08.481382 16362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45829 (error 108)
I20260619 14:46:08.482566 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 16569
I20260619 14:46:08.483675 16870 ts_tablet_manager.cc:1905] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:08.484565 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:08.484255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45829 (local address 127.0.0.1:37658)
0619 14:46:08.484397 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:08.484470 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 14:46:08.484777 16362 catalog_manager.cc:4584] TS 8504541fa3b84b0da8e0376ff83c6040 (127.0.98.1:45829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ea055d409744ea695a6ddec2687ebc7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45829: connect: Connection refused (error 111)
I20260619 14:46:08.486552 16870 ts_tablet_manager.cc:1918] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:08.486651 16870 log.cc:1196] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/7ea055d409744ea695a6ddec2687ebc7
I20260619 14:46:08.486985 16870 ts_tablet_manager.cc:1939] T 7ea055d409744ea695a6ddec2687ebc7 P e416f9f3d8104e42826ed1005c478b30: Deleting consensus metadata
I20260619 14:46:08.488581 16363 catalog_manager.cc:4841] TS e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975): tablet 7ea055d409744ea695a6ddec2687ebc7 (table test-table [id=07500dab3fd54f3b8f9ce0bee0918a0d]) successfully deleted
I20260619 14:46:08.489120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 16706
I20260619 14:46:08.495162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 16344
I20260619 14:46:28.504689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45837
--webserver_interface=127.0.98.62
--webserver_port=46491
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:28.616762 16873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:28.617057 16873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:28.617101 16873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:28.621484 16873 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 14:46:28.621555 16873 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 14:46:28.621572 16873 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 14:46:28.621587 16873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:28.621605 16873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:28.621624 16873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:28.621640 16873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:28.621690 16873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:28.627187 16873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:28.628373 16873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:28.629513 16873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:28.638449 16881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:28.638883 16879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:28.638927 16878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:28.639098 16873 server_base.cc:1034] running on GCE node
I20260619 14:46:28.639846 16873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:28.641094 16873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:28.642380 16873 hybrid_clock.cc:648] HybridClock initialized: now 1781880388642345 us; error 50 us; skew 500 ppm
Jun 19 14:46:28 dist-test-slave-cr22 krb5kdc[15796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:28.651026 16873 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:46:28.652196 16873 webserver.cc:460] Webserver started at http://127.0.98.62:46491/ using document root <none> and password file <none>
I20260619 14:46:28.652832 16873 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:28.652932 16873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:28.657125 16873 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:46:28.659896 16888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:28.661511 16873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:46:28.661669 16873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "1fea269cd46749f88370613b756334a2"
format_stamp: "Formatted at 2026-06-19 14:45:46 on dist-test-slave-cr22"
server_key: "36403746de8bf71fb0570e6de8c709ec"
server_key_iv: "db02aae0cd6eec9b2ac4cd9d391b0227"
server_key_version: "encryptionkey@0"
I20260619 14:46:28.662248 16873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:28.679601 16873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:28.682847 16873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:28.683079 16873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:28.690574 16873 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45837
I20260619 14:46:28.691066 16940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45837 every 8 connection(s)
I20260619 14:46:28.691854 16873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:28.692170 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 16873
I20260619 14:46:28.692687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45829
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45837
--webserver_port=37193
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:46:28.697247 16941 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:46:28.698974 16941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: Bootstrap starting.
I20260619 14:46:28.717013 16941 log.cc:824] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:28.728788 16941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:46:28.729225 16941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: Bootstrap complete.
I20260619 14:46:28.733350 16941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fea269cd46749f88370613b756334a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45837 } }
I20260619 14:46:28.733750 16941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fea269cd46749f88370613b756334a2, State: Initialized, Role: FOLLOWER
I20260619 14:46:28.734241 16941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fea269cd46749f88370613b756334a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45837 } }
I20260619 14:46:28.734355 16941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:28.734449 16941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:28.734571 16941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:46:28.736641 16941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fea269cd46749f88370613b756334a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45837 } }
I20260619 14:46:28.736992 16941 leader_election.cc:302] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fea269cd46749f88370613b756334a2; no voters:
I20260619 14:46:28.737339 16941 leader_election.cc:288] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 14:46:28.737424 16945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 14:46:28.737674 16945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [term 3 LEADER]: Becoming Leader. State: Replica: 1fea269cd46749f88370613b756334a2, State: Running, Role: LEADER
I20260619 14:46:28.738034 16945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fea269cd46749f88370613b756334a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45837 } }
I20260619 14:46:28.738618 16941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:28.738989 16945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fea269cd46749f88370613b756334a2. Latest consensus state: current_term: 3 leader_uuid: "1fea269cd46749f88370613b756334a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fea269cd46749f88370613b756334a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45837 } } }
I20260619 14:46:28.739503 16945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:28.740367 16946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1fea269cd46749f88370613b756334a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fea269cd46749f88370613b756334a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45837 } } }
I20260619 14:46:28.740483 16946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:28.744063 16961 catalog_manager.cc:1259] Loaded cluster ID: e5217a81c1d341539d09a86632200d86
I20260619 14:46:28.744129 16961 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: loading cluster ID for follower catalog manager: success
I20260619 14:46:28.746596 16961 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1fea269cd46749f88370613b756334a2: acquiring CA information for follower catalog manager: success
F20260619 14:46:28.747854 16961 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe9cd9639ad google::LogMessage::Fail() at ??:0
@ 0x7fe9cd96688a google::LogMessage::SendToLog() at ??:0
@ 0x7fe9cd9634de google::LogMessage::Flush() at ??:0
@ 0x7fe9cd964a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe9d062718d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe9d82c8989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fe9d82a5023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fe9d82a554b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fe9d829e652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fe9d829d9e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7fe9d82e4541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe9cfba6512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe9d11ea6db start_thread at ??:0
@ 0x7fe9cc46161f clone at ??:0
W20260619 14:46:28.833457 16943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:28.833784 16943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:28.833865 16943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:28.838177 16943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:28.838284 16943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:28.838373 16943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:28.844064 16943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37193
--tserver_master_addrs=127.0.98.62:45837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:28.845235 16943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:28.846302 16943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:28.857263 16981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:28.857506 16984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:28.857277 16982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:28.858839 16943 server_base.cc:1034] running on GCE node
I20260619 14:46:28.859586 16943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:28.860311 16943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:28.861537 16943 hybrid_clock.cc:648] HybridClock initialized: now 1781880388861518 us; error 44 us; skew 500 ppm
Jun 19 14:46:28 dist-test-slave-cr22 krb5kdc[15796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:28.865767 16943 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:46:28.866875 16943 webserver.cc:460] Webserver started at http://127.0.98.1:37193/ using document root <none> and password file <none>
I20260619 14:46:28.867587 16943 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:28.867691 16943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:28.872107 16943 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:46:28.875495 16991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:28.876963 16943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:46:28.877116 16943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "8504541fa3b84b0da8e0376ff83c6040"
format_stamp: "Formatted at 2026-06-19 14:45:46 on dist-test-slave-cr22"
server_key: "b7b114f20f265bbc27b5dacd35bde55e"
server_key_iv: "0db6786587a3a9b4dd80179df0cd0394"
server_key_version: "encryptionkey@0"
I20260619 14:46:28.877728 16943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:28.892858 16943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:28.897301 16943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:28.897532 16943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:28.898175 16943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:28.899406 16998 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:46:28.905426 16943 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:46:28.905553 16943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 14:46:28.905638 16943 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:46:28.907635 16943 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:46:28.907709 16943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260619 14:46:28.908008 16998 tablet_bootstrap.cc:492] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040: Bootstrap starting.
I20260619 14:46:28.918679 16943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45829
I20260619 14:46:28.918728 17105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45829 every 8 connection(s)
I20260619 14:46:28.919771 16943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:28.920377 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 16943
I20260619 14:46:28.920915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39519
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45837
--webserver_port=41755
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:46:28.930212 16998 log.cc:824] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:28.937382 16998 tablet_bootstrap.cc:492] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:46:28.937798 16998 tablet_bootstrap.cc:492] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040: Bootstrap complete.
I20260619 14:46:28.938902 16998 ts_tablet_manager.cc:1397] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.011s
I20260619 14:46:28.942279 16998 raft_consensus.cc:348] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:46:28.943081 16998 raft_consensus.cc:729] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8504541fa3b84b0da8e0376ff83c6040, State: Initialized, Role: FOLLOWER
I20260619 14:46:28.943699 16998 consensus_queue.cc:260] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:46:28.944806 16998 ts_tablet_manager.cc:1428] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040: Time spent starting tablet: real 0.006s user 0.009s sys 0.000s
W20260619 14:46:29.001838 17109 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:46:28.923126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:37467)
0619 14:46:28.923794 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:28.923866 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:28.925075 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:29.001004 (+ 75929us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":194,"threads_started":1}
W20260619 14:46:29.002398 17106 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45837 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45837: Client connection negotiation failed: client connection to 127.0.98.62:45837: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:46:29.003165 17109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:29.002904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:55423)
0619 14:46:29.003041 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:29.003106 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 14:46:29.004022 17109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:29.003769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:52857)
0619 14:46:29.003928 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:29.003972 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260619 14:46:29.004206 17106 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:46:29.064677 17108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:29.065001 17108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:29.065088 17108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:29.069574 17108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:29.069689 17108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:29.069818 17108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:29.076426 17108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41755
--tserver_master_addrs=127.0.98.62:45837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:29.077592 17108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:29.078697 17108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:29.089071 17119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:29.089221 17116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:29.089159 17117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:29.089337 17108 server_base.cc:1034] running on GCE node
I20260619 14:46:29.090188 17108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:29.091007 17108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:29.092240 17108 hybrid_clock.cc:648] HybridClock initialized: now 1781880389092221 us; error 34 us; skew 500 ppm
Jun 19 14:46:29 dist-test-slave-cr22 krb5kdc[15796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:29.095556 17108 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:46:29.096565 17108 webserver.cc:460] Webserver started at http://127.0.98.2:41755/ using document root <none> and password file <none>
I20260619 14:46:29.097213 17108 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:29.097323 17108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:29.101667 17108 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:46:29.104417 17126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:29.105901 17108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:29.106070 17108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "411bce22959b49a88031e65d983d5244"
format_stamp: "Formatted at 2026-06-19 14:45:46 on dist-test-slave-cr22"
server_key: "9db852459ecdfdf8da62abe158911c92"
server_key_iv: "159bcfcd24e32b3d804007c0c93f63bb"
server_key_version: "encryptionkey@0"
I20260619 14:46:29.106725 17108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:29.126163 17108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:29.130236 17108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:29.130551 17108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:29.131212 17108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:29.132131 17108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:29.132212 17108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:29.132293 17108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:29.132349 17108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:29.142622 17108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39519
I20260619 14:46:29.142647 17239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39519 every 8 connection(s)
I20260619 14:46:29.143723 17108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:29.148655 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:29.146062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:50495)
0619 14:46:29.147299 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:29.147761 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":552,"threads_started":1}
W20260619 14:46:29.149492 17240 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45837 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45837: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
I20260619 14:46:29.150302 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:29.149973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:44185)
0619 14:46:29.150120 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:29.150226 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 14:46:29.150596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 17108
I20260619 14:46:29.151116 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35975
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45837
--webserver_port=41801
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:46:29.151242 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:29.151000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:37781)
0619 14:46:29.151119 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:29.151174 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 14:46:29.151496 17240 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:46:29.265218 17243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:29.265676 17243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:29.265767 17243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:29.270604 17243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:29.270691 17243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:29.270818 17243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:29.276768 17243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41801
--tserver_master_addrs=127.0.98.62:45837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:29.277906 17243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:29.278939 17243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:29.288955 17249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:29.288955 17248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:29.288950 17251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:29.289364 17243 server_base.cc:1034] running on GCE node
I20260619 14:46:29.289754 17243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:29.290439 17243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:29.291558 17243 hybrid_clock.cc:648] HybridClock initialized: now 1781880389291538 us; error 31 us; skew 500 ppm
Jun 19 14:46:29 dist-test-slave-cr22 krb5kdc[15796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:29.294685 17243 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:46:29.295691 17243 webserver.cc:460] Webserver started at http://127.0.98.3:41801/ using document root <none> and password file <none>
I20260619 14:46:29.296281 17243 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:29.296342 17243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:29.300259 17243 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:46:29.302500 17258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:29.303884 17243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:29.304025 17243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "e416f9f3d8104e42826ed1005c478b30"
format_stamp: "Formatted at 2026-06-19 14:45:47 on dist-test-slave-cr22"
server_key: "7fa2191f19e9ad0a012a4e6ba7085737"
server_key_iv: "c1933ecfc8e74392b51c5ab5fb9010e6"
server_key_version: "encryptionkey@0"
I20260619 14:46:29.304625 17243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:29.321025 17243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:29.324529 17243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:29.324747 17243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:29.325371 17243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:29.326236 17243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:29.326310 17243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:29.326381 17243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:29.326427 17243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:29.335898 17243 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35975
I20260619 14:46:29.335918 17371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35975 every 8 connection(s)
I20260619 14:46:29.336964 17243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:29.338572 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 17243
I20260619 14:46:29.340921 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:29.339273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.3:40307)
0619 14:46:29.340090 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:29.340212 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":187,"threads_started":1}
W20260619 14:46:29.341343 17372 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45837 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45837: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
I20260619 14:46:29.341944 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:29.341716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.3:42555)
0619 14:46:29.341836 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:29.341891 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 14:46:29.342854 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:29.342578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.3:43373)
0619 14:46:29.342754 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:29.342800 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 14:46:29.343065 17372 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 14:46:30.005613 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.005051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:42007)
0619 14:46:30.005404 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:30.005472 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":133,"threads_started":1}
I20260619 14:46:30.152701 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.152180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:39371)
0619 14:46:30.152481 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:30.152541 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":79,"threads_started":1}
I20260619 14:46:30.344213 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.343772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.3:36599)
0619 14:46:30.344041 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:30.344100 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
I20260619 14:46:30.450409 17379 raft_consensus.cc:482] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:30.450649 17379 raft_consensus.cc:504] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:46:30.452101 17379 leader_election.cc:288] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975), 411bce22959b49a88031e65d983d5244 (127.0.98.2:39519)
Jun 19 14:46:30 dist-test-slave-cr22 krb5kdc[15796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 19 14:46:30 dist-test-slave-cr22 krb5kdc[15796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260619 14:46:30.469348 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.452683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44823 (local address 127.0.98.2:39519)
0619 14:46:30.453074 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:30.453093 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:30.453139 (+ 46us) server_negotiation.cc:407] Connection header received
0619 14:46:30.454024 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:30.454041 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:30.454367 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:30.454770 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:30.455987 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.457193 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.458342 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.459053 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.459280 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.462556 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:30.462561 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:30.462575 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:30.462606 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:30.465237 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:30.466411 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:30.466416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:30.466420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:30.466608 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:30.467180 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:30.467183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:30.467185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:30.467823 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:30.467972 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:30.468834 (+ 862us) server_negotiation.cc:299] Negotiation successful
0619 14:46:30.469147 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":73,"threads_started":1}
I20260619 14:46:30.469370 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.452526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33993 (local address 127.0.98.3:35975)
0619 14:46:30.452890 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:30.452915 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:30.452962 (+ 47us) server_negotiation.cc:407] Connection header received
0619 14:46:30.453824 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:30.453830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:30.454200 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:30.454477 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:30.455988 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.456980 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.458518 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.459021 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.459583 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.462825 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:30.462833 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:30.462850 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:30.462887 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:30.465229 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:30.466393 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:30.466401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:30.466407 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:30.466608 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:30.467180 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:30.467184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:30.467185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:30.467823 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:30.467972 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:30.468871 (+ 899us) server_negotiation.cc:299] Negotiation successful
0619 14:46:30.469147 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":142,"threads_started":1}
I20260619 14:46:30.469399 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.452316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35975 (local address 127.0.98.1:33993)
0619 14:46:30.452466 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:30.452483 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:30.452592 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:30.454677 (+ 2085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:30.454684 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:30.455240 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:30.455784 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.455800 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.457158 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.457162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.458364 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.458376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.459210 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.459214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.459573 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.460448 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:30.460460 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:30.462615 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:30.465558 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:30.465567 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:30.465585 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:30.466176 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:30.466815 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:30.466818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:30.466819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:30.467039 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:30.467973 (+ 934us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:30.467981 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:30.468347 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 14:46:30.468598 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 14:46:30.469015 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":267,"spinlock_wait_cycles":26752}
I20260619 14:46:30.469456 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.452450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39519 (local address 127.0.98.1:44823)
0619 14:46:30.452809 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:30.452822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:30.452914 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:30.454739 (+ 1825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:30.454743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:30.455387 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:30.455783 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.455800 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.457349 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.457352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.458211 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.458221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.459227 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.459230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.459522 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.460405 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:30.460423 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:30.462240 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:30.465558 (+ 3318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:30.465567 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:30.465585 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:30.466281 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:30.466781 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:30.466784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:30.466789 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:30.467039 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:30.467973 (+ 934us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:30.467981 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:30.468347 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 14:46:30.468597 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 14:46:30.469069 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":220,"spinlock_wait_cycles":23680,"thread_start_us":85,"threads_started":1}
W20260619 14:46:30.472234 16992 leader_election.cc:341] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 411bce22959b49a88031e65d983d5244 (127.0.98.2:39519): Not found: Tablet not found: 7ea055d409744ea695a6ddec2687ebc7
W20260619 14:46:30.472450 16994 leader_election.cc:341] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975): Not found: Tablet not found: 7ea055d409744ea695a6ddec2687ebc7
I20260619 14:46:30.472555 16994 leader_election.cc:302] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8504541fa3b84b0da8e0376ff83c6040; no voters: 411bce22959b49a88031e65d983d5244, e416f9f3d8104e42826ed1005c478b30
I20260619 14:46:30.472959 17379 raft_consensus.cc:2726] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:46:31.006880 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:31.006424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:45609)
0619 14:46:31.006713 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:31.006771 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
I20260619 14:46:31.154089 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:31.153666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:50851)
0619 14:46:31.153935 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:31.153995 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
I20260619 14:46:31.345471 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:31.345064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.3:50603)
0619 14:46:31.345314 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:31.345381 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
I20260619 14:46:32.008227 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:32.007773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:33075)
0619 14:46:32.008069 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:32.008129 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":87,"threads_started":1}
I20260619 14:46:32.155571 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:32.155039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:54495)
0619 14:46:32.155325 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:32.155395 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":83,"threads_started":1}
I20260619 14:46:32.346887 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:32.346423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.3:36569)
0619 14:46:32.346705 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:32.346774 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
I20260619 14:46:32.619392 17389 raft_consensus.cc:482] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:32.619601 17389 raft_consensus.cc:504] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:46:32.620102 17389 leader_election.cc:288] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975), 411bce22959b49a88031e65d983d5244 (127.0.98.2:39519)
W20260619 14:46:32.621202 16994 leader_election.cc:341] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975): Not found: Tablet not found: 7ea055d409744ea695a6ddec2687ebc7
W20260619 14:46:32.621306 16992 leader_election.cc:341] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 411bce22959b49a88031e65d983d5244 (127.0.98.2:39519): Not found: Tablet not found: 7ea055d409744ea695a6ddec2687ebc7
I20260619 14:46:32.621470 16992 leader_election.cc:302] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8504541fa3b84b0da8e0376ff83c6040; no voters: 411bce22959b49a88031e65d983d5244, e416f9f3d8104e42826ed1005c478b30
I20260619 14:46:32.621670 17389 raft_consensus.cc:2726] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:46:33.009471 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.009049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:38945)
0619 14:46:33.009318 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:33.009375 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":73,"threads_started":1}
I20260619 14:46:33.156766 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.156299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:40691)
0619 14:46:33.156587 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:33.156663 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":77,"threads_started":1}
I20260619 14:46:33.348282 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.347852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.3:40519)
0619 14:46:33.348133 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:33.348191 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":117,"threads_started":1}
I20260619 14:46:34.011302 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:34.010653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:48285)
0619 14:46:34.011034 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:34.011097 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":97,"threads_started":1}
I20260619 14:46:34.158454 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:34.157943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:37777)
0619 14:46:34.158279 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:34.158352 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":86,"threads_started":1}
I20260619 14:46:34.349850 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:34.349247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.3:52221)
0619 14:46:34.349627 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:34.349738 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":158,"threads_started":1}
I20260619 14:46:34.559729 17396 raft_consensus.cc:482] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:34.559931 17396 raft_consensus.cc:504] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:46:34.561069 17396 leader_election.cc:288] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975), 411bce22959b49a88031e65d983d5244 (127.0.98.2:39519)
W20260619 14:46:34.561654 16994 leader_election.cc:341] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975): Not found: Tablet not found: 7ea055d409744ea695a6ddec2687ebc7
W20260619 14:46:34.561856 16992 leader_election.cc:341] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 411bce22959b49a88031e65d983d5244 (127.0.98.2:39519): Not found: Tablet not found: 7ea055d409744ea695a6ddec2687ebc7
I20260619 14:46:34.561923 16992 leader_election.cc:302] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8504541fa3b84b0da8e0376ff83c6040; no voters: 411bce22959b49a88031e65d983d5244, e416f9f3d8104e42826ed1005c478b30
I20260619 14:46:34.562083 17396 raft_consensus.cc:2726] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:46:35.012748 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:35.012277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:39867)
0619 14:46:35.012574 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:35.012647 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260619 14:46:35.159935 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:35.159486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:43959)
0619 14:46:35.159772 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:35.159835 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":101,"threads_started":1}
I20260619 14:46:35.351377 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:35.350887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.3:38355)
0619 14:46:35.351160 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:35.351215 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":93,"threads_started":1}
I20260619 14:46:36.014037 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:36.013597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:50295)
0619 14:46:36.013875 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:36.013940 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":84,"threads_started":1}
I20260619 14:46:36.161203 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:36.160739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:43303)
0619 14:46:36.161048 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:36.161106 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":74,"threads_started":1}
I20260619 14:46:36.352948 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:36.352330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.3:43959)
0619 14:46:36.352770 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:36.352849 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":95,"threads_started":1}
I20260619 14:46:36.806041 17403 raft_consensus.cc:482] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:36.806180 17403 raft_consensus.cc:504] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:46:36.806789 17403 leader_election.cc:288] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975), 411bce22959b49a88031e65d983d5244 (127.0.98.2:39519)
W20260619 14:46:36.807801 16992 leader_election.cc:341] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 411bce22959b49a88031e65d983d5244 (127.0.98.2:39519): Not found: Tablet not found: 7ea055d409744ea695a6ddec2687ebc7
W20260619 14:46:36.807915 16994 leader_election.cc:341] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975): Not found: Tablet not found: 7ea055d409744ea695a6ddec2687ebc7
I20260619 14:46:36.808019 16994 leader_election.cc:302] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8504541fa3b84b0da8e0376ff83c6040; no voters: 411bce22959b49a88031e65d983d5244, e416f9f3d8104e42826ed1005c478b30
I20260619 14:46:36.808184 17403 raft_consensus.cc:2726] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:46:37.015377 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:37.014887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:58329)
0619 14:46:37.015165 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:37.015236 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20260619 14:46:37.162740 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:37.162266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:54723)
0619 14:46:37.162566 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:37.162628 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":97,"threads_started":1}
I20260619 14:46:37.354395 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:37.353935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.3:35407)
0619 14:46:37.354219 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:37.354284 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":77,"threads_started":1}
I20260619 14:46:38.016821 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:38.016327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:50367)
0619 14:46:38.016656 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:38.016725 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":86,"threads_started":1}
I20260619 14:46:38.164290 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:38.163825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:42961)
0619 14:46:38.164116 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:38.164178 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":79,"threads_started":1}
I20260619 14:46:38.355803 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:38.355337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.3:48349)
0619 14:46:38.355645 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:38.355704 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":147,"threads_started":1}
I20260619 14:46:39.018373 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:39.017909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:53683)
0619 14:46:39.018220 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:39.018277 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":78,"threads_started":1}
I20260619 14:46:39.165632 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:39.165149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:34101)
0619 14:46:39.165470 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:39.165529 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":123,"threads_started":1}
I20260619 14:46:39.263046 17412 raft_consensus.cc:482] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:39.263188 17412 raft_consensus.cc:504] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:46:39.263738 17412 leader_election.cc:288] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975), 411bce22959b49a88031e65d983d5244 (127.0.98.2:39519)
W20260619 14:46:39.264601 16994 leader_election.cc:341] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975): Not found: Tablet not found: 7ea055d409744ea695a6ddec2687ebc7
W20260619 14:46:39.264686 16992 leader_election.cc:341] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 411bce22959b49a88031e65d983d5244 (127.0.98.2:39519): Not found: Tablet not found: 7ea055d409744ea695a6ddec2687ebc7
I20260619 14:46:39.264730 16994 leader_election.cc:302] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8504541fa3b84b0da8e0376ff83c6040; no voters: 411bce22959b49a88031e65d983d5244, e416f9f3d8104e42826ed1005c478b30
I20260619 14:46:39.264927 17412 raft_consensus.cc:2726] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:46:39.357139 17413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:39.356707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.3:35145)
0619 14:46:39.356980 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:39.357033 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":75,"threads_started":1}
I20260619 14:46:40.019865 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:40.019338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:53847)
0619 14:46:40.019703 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:40.019766 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":136,"threads_started":1}
I20260619 14:46:40.167014 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:40.166473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:36617)
0619 14:46:40.166813 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:40.166886 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":94,"threads_started":1}
I20260619 14:46:40.358515 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:40.358124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.3:51969)
0619 14:46:40.358369 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:40.358425 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
I20260619 14:46:41.021247 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:41.020774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:60897)
0619 14:46:41.021082 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:41.021146 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":78,"threads_started":1}
I20260619 14:46:41.168380 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:41.167938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:35123)
0619 14:46:41.168220 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:41.168278 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":79,"threads_started":1}
I20260619 14:46:41.201998 17419 raft_consensus.cc:482] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:41.202126 17419 raft_consensus.cc:504] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e416f9f3d8104e42826ed1005c478b30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35975 } } peers { permanent_uuid: "411bce22959b49a88031e65d983d5244" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39519 } } peers { permanent_uuid: "8504541fa3b84b0da8e0376ff83c6040" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45829 } }
I20260619 14:46:41.202487 17419 leader_election.cc:288] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975), 411bce22959b49a88031e65d983d5244 (127.0.98.2:39519)
W20260619 14:46:41.203570 16992 leader_election.cc:341] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 411bce22959b49a88031e65d983d5244 (127.0.98.2:39519): Not found: Tablet not found: 7ea055d409744ea695a6ddec2687ebc7
W20260619 14:46:41.203685 16994 leader_election.cc:341] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e416f9f3d8104e42826ed1005c478b30 (127.0.98.3:35975): Not found: Tablet not found: 7ea055d409744ea695a6ddec2687ebc7
I20260619 14:46:41.203754 16994 leader_election.cc:302] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8504541fa3b84b0da8e0376ff83c6040; no voters: 411bce22959b49a88031e65d983d5244, e416f9f3d8104e42826ed1005c478b30
I20260619 14:46:41.203933 17419 raft_consensus.cc:2726] T 7ea055d409744ea695a6ddec2687ebc7 P 8504541fa3b84b0da8e0376ff83c6040 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:46:41.360000 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:41.359574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.3:53917)
0619 14:46:41.359843 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:41.359905 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260619 14:46:42.022608 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:42.022200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:40423)
0619 14:46:42.022454 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:42.022511 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
I20260619 14:46:42.169656 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:42.169208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:46443)
0619 14:46:42.169497 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:42.169557 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":80,"threads_started":1}
I20260619 14:46:42.361200 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:42.360790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.3:56609)
0619 14:46:42.361044 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:42.361100 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260619 14:46:43.024135 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:43.023513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:45205)
0619 14:46:43.023985 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:43.024039 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":224,"threads_started":1}
I20260619 14:46:43.170960 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:43.170538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:48261)
0619 14:46:43.170805 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:43.170862 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
I20260619 14:46:43.362645 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:43.362129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.3:42591)
0619 14:46:43.362486 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:43.362548 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":99,"threads_started":1}
I20260619 14:46:44.025869 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:44.025368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.1:40619)
0619 14:46:44.025624 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:44.025716 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":88,"threads_started":1}
I20260619 14:46:44.172508 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:44.172078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45837 (local address 127.0.98.2:50339)
0619 14:46:44.172357 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:44.172415 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":96,"threads_started":1}
I20260619 14:46:44.345656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 16943
I20260619 14:46:44.351893 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 17108
I20260619 14:46:44.357389 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 17243
2026-06-19T14:46:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59294 ms)
[----------] 30 tests from SecurityITest (249091 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:46:44 dist-test-slave-cr22 krb5kdc[17431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:46:44 dist-test-slave-cr22 krb5kdc[17431](info): set up 2 sockets
Jun 19 14:46:44 dist-test-slave-cr22 krb5kdc[17431](info): commencing operation
krb5kdc: starting...
W20260619 14:46:45.475689 392 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:46:45 dist-test-slave-cr22 krb5kdc[17431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:46:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:46:45.611641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42789
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:45.724913 17444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:45.725195 17444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:45.725251 17444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:45.729429 17444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:45.729494 17444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:45.729514 17444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:45.729532 17444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:45.729547 17444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:45.734930 17444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:45.736037 17444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:45.737064 17444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:45.746132 17449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:45.746140 17452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:45.746145 17450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:45.746284 17444 server_base.cc:1034] running on GCE node
I20260619 14:46:45.747023 17444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:45.748274 17444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:45.749495 17444 hybrid_clock.cc:648] HybridClock initialized: now 1781880405749466 us; error 42 us; skew 500 ppm
Jun 19 14:46:45 dist-test-slave-cr22 krb5kdc[17431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:45.752693 17444 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:46:45.753633 17444 webserver.cc:460] Webserver started at http://127.0.98.62:45755/ using document root <none> and password file <none>
I20260619 14:46:45.754204 17444 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:45.754305 17444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:45.754546 17444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:45.756803 17444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "8e637c75b0354d8f8f210e11b1518fab"
format_stamp: "Formatted at 2026-06-19 14:46:45 on dist-test-slave-cr22"
server_key: "e3d81833dec42112f34528fdecacf396"
server_key_iv: "e3dee1914fa61d7bf8655daa4ea4d58f"
server_key_version: "encryptionkey@0"
I20260619 14:46:45.757376 17444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "8e637c75b0354d8f8f210e11b1518fab"
format_stamp: "Formatted at 2026-06-19 14:46:45 on dist-test-slave-cr22"
server_key: "e3d81833dec42112f34528fdecacf396"
server_key_iv: "e3dee1914fa61d7bf8655daa4ea4d58f"
server_key_version: "encryptionkey@0"
I20260619 14:46:45.761155 17444 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:46:45.763949 17459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:45.765189 17444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:46:45.765336 17444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "8e637c75b0354d8f8f210e11b1518fab"
format_stamp: "Formatted at 2026-06-19 14:46:45 on dist-test-slave-cr22"
server_key: "e3d81833dec42112f34528fdecacf396"
server_key_iv: "e3dee1914fa61d7bf8655daa4ea4d58f"
server_key_version: "encryptionkey@0"
I20260619 14:46:45.765457 17444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:45.776094 17444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:45.779407 17444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:45.779656 17444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:45.786753 17444 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42789
I20260619 14:46:45.786782 17511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42789 every 8 connection(s)
I20260619 14:46:45.787706 17444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:45.788450 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 17444
I20260619 14:46:45.788569 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:45.788905 392 external_mini_cluster.cc:1428] Setting key c9f23219f4ee0b38d96f02d7c686d9bc
I20260619 14:46:45.790942 17512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:45.796939 17512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab: Bootstrap starting.
Jun 19 14:46:45 dist-test-slave-cr22 krb5kdc[17431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:45.799507 17512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:45.800227 17512 log.cc:824] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:45.802120 17512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab: No bootstrap required, opened a new log
I20260619 14:46:45.802853 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:45.790313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36284 (local address 127.0.98.62:42789)
0619 14:46:45.790750 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:45.790764 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:45.790796 (+ 32us) server_negotiation.cc:407] Connection header received
0619 14:46:45.791550 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:45.791568 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:45.791883 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:45.792219 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:45.793003 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:45.794007 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:45.794865 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:45.795329 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:45.795521 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:45.798264 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:45.798269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:45.798281 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:45.798309 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:45.800057 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:45.800590 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:45.800601 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:45.800605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:45.800712 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:45.800995 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:45.801000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:45.801001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:45.801387 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:45.801596 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:45.801831 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 14:46:45.802115 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":148,"threads_started":1}
I20260619 14:46:45.806080 17512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e637c75b0354d8f8f210e11b1518fab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42789 } }
I20260619 14:46:45.806360 17512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:45.806442 17512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e637c75b0354d8f8f210e11b1518fab, State: Initialized, Role: FOLLOWER
I20260619 14:46:45.806838 17512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e637c75b0354d8f8f210e11b1518fab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42789 } }
I20260619 14:46:45.806983 17512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:45.807067 17512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:45.807184 17512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:45.808112 17512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e637c75b0354d8f8f210e11b1518fab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42789 } }
I20260619 14:46:45.808435 17512 leader_election.cc:302] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e637c75b0354d8f8f210e11b1518fab; no voters:
I20260619 14:46:45.808781 17512 leader_election.cc:288] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:45.808945 17517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:45.809243 17517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab [term 1 LEADER]: Becoming Leader. State: Replica: 8e637c75b0354d8f8f210e11b1518fab, State: Running, Role: LEADER
I20260619 14:46:45.809525 17517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e637c75b0354d8f8f210e11b1518fab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42789 } }
I20260619 14:46:45.809980 17512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:45.810542 17517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e637c75b0354d8f8f210e11b1518fab. Latest consensus state: current_term: 1 leader_uuid: "8e637c75b0354d8f8f210e11b1518fab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e637c75b0354d8f8f210e11b1518fab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42789 } } }
I20260619 14:46:45.810659 17517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:45.811060 17522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:45.811115 17518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e637c75b0354d8f8f210e11b1518fab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e637c75b0354d8f8f210e11b1518fab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42789 } } }
I20260619 14:46:45.811296 17518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:45.814087 17522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:45.820022 17522 catalog_manager.cc:1347] Generated new cluster ID: 37997aaf0e2e42e1b66169f1b1644bdf
I20260619 14:46:45.820106 17522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:45.848517 17522 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:45.849531 17522 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:45.855707 17522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab: Generated new TSK 0
I20260619 14:46:45.856592 17522 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:46:45.899302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42789
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:46.017046 17538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:46.017405 17538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:46.017488 17538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:46.022261 17538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:46.022392 17538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:46.022500 17538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:46.028811 17538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:46.030104 17538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:46.031530 17538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:46.044070 17546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:46.044070 17543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:46.044073 17544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:46.044389 17538 server_base.cc:1034] running on GCE node
I20260619 14:46:46.045342 17538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:46.046145 17538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:46.047546 17538 hybrid_clock.cc:648] HybridClock initialized: now 1781880406047514 us; error 57 us; skew 500 ppm
Jun 19 14:46:46 dist-test-slave-cr22 krb5kdc[17431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:46.052299 17538 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:46:46.053448 17538 webserver.cc:460] Webserver started at http://127.0.98.1:40603/ using document root <none> and password file <none>
I20260619 14:46:46.054091 17538 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:46.054203 17538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:46.054459 17538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:46.056823 17538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "7fc19052081943108a860d2fda03b646"
format_stamp: "Formatted at 2026-06-19 14:46:46 on dist-test-slave-cr22"
server_key: "8c568ac51facb591845db93e083fd214"
server_key_iv: "9316a0b5df2e75d66666fe0e5dfba200"
server_key_version: "encryptionkey@0"
I20260619 14:46:46.057464 17538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7fc19052081943108a860d2fda03b646"
format_stamp: "Formatted at 2026-06-19 14:46:46 on dist-test-slave-cr22"
server_key: "8c568ac51facb591845db93e083fd214"
server_key_iv: "9316a0b5df2e75d66666fe0e5dfba200"
server_key_version: "encryptionkey@0"
I20260619 14:46:46.061743 17538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:46:46.064594 17553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:46.066053 17538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:46.066150 17538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "7fc19052081943108a860d2fda03b646"
format_stamp: "Formatted at 2026-06-19 14:46:46 on dist-test-slave-cr22"
server_key: "8c568ac51facb591845db93e083fd214"
server_key_iv: "9316a0b5df2e75d66666fe0e5dfba200"
server_key_version: "encryptionkey@0"
I20260619 14:46:46.066277 17538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:46.107112 17538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:46.110811 17538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:46.111034 17538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:46.111734 17538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:46.112663 17538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:46.112712 17538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:46.112778 17538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:46.112815 17538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:46.122538 17538 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42433
I20260619 14:46:46.122560 17666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42433 every 8 connection(s)
I20260619 14:46:46.123624 17538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:46.127204 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 17538
I20260619 14:46:46.127310 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:46:46.127646 392 external_mini_cluster.cc:1428] Setting key a67ca0ef35869fbbae7793142215f83e
Jun 19 14:46:46 dist-test-slave-cr22 krb5kdc[17431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:46:46.141422 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.125632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45621 (local address 127.0.98.62:42789)
0619 14:46:46.125823 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:46.125827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:46.126758 (+ 931us) server_negotiation.cc:407] Connection header received
0619 14:46:46.128934 (+ 2176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:46.128937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:46.129014 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:46.129101 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:46.130940 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.132002 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.133247 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.133743 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.133888 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.136568 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:46.136576 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:46.136581 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:46.136618 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:46.138536 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:46.139192 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:46.139196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:46.139197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:46.139304 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:46.139825 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:46.139828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:46.139829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:46.140163 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:46.140261 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:46.141015 (+ 754us) server_negotiation.cc:299] Negotiation successful
0619 14:46:46.141228 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 14:46:46.142194 17669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.126011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42789 (local address 127.0.98.1:45621)
0619 14:46:46.126555 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:46.126601 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:46.128235 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:46.129392 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:46.129402 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:46.129966 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:46.130689 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.130712 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.132163 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.132166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.133099 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.133110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.133942 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.133945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.134090 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.134841 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:46.134858 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:46.136377 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:46.138730 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:46.138735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:46.138755 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:46.139042 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:46.139492 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:46.139496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:46.139499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:46.139689 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:46.140455 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:46.140461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:46.140727 (+ 266us) client_negotiation.cc:769] Sending connection context
0619 14:46:46.140897 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 14:46:46.141194 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":135,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:46:46.143711 17667 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42789
I20260619 14:46:46.144027 17667 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:46.144636 17667 heartbeater.cc:511] Master 127.0.98.62:42789 requested a full tablet report, sending...
I20260619 14:46:46.146404 17476 ts_manager.cc:194] Registered new tserver with Master: 7fc19052081943108a860d2fda03b646 (127.0.98.1:42433)
I20260619 14:46:46.147794 17476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:46:46.159864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42789
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:46.285732 17672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:46.286026 17672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:46.286108 17672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:46.290771 17672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:46.290843 17672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:46.290946 17672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:46.296762 17672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:46.297946 17672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:46.298993 17672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:46.309067 17677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:46.309078 17678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:46.309087 17680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:46.309471 17672 server_base.cc:1034] running on GCE node
I20260619 14:46:46.310001 17672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:46.310662 17672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:46.311825 17672 hybrid_clock.cc:648] HybridClock initialized: now 1781880406311780 us; error 57 us; skew 500 ppm
Jun 19 14:46:46 dist-test-slave-cr22 krb5kdc[17431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:46.314913 17672 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:46:46.316066 17672 webserver.cc:460] Webserver started at http://127.0.98.2:41267/ using document root <none> and password file <none>
I20260619 14:46:46.316704 17672 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:46.316767 17672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:46.316994 17672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:46.319031 17672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "89755e1bdae84acc8b22acb7bcbfa1a9"
format_stamp: "Formatted at 2026-06-19 14:46:46 on dist-test-slave-cr22"
server_key: "d6fd83c34bdef520e3cc6cea62d47005"
server_key_iv: "dbc1251b26719ec255097f824d7b1da7"
server_key_version: "encryptionkey@0"
I20260619 14:46:46.319635 17672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "89755e1bdae84acc8b22acb7bcbfa1a9"
format_stamp: "Formatted at 2026-06-19 14:46:46 on dist-test-slave-cr22"
server_key: "d6fd83c34bdef520e3cc6cea62d47005"
server_key_iv: "dbc1251b26719ec255097f824d7b1da7"
server_key_version: "encryptionkey@0"
I20260619 14:46:46.323411 17672 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:46:46.325994 17687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:46.327370 17672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:46:46.327566 17672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "89755e1bdae84acc8b22acb7bcbfa1a9"
format_stamp: "Formatted at 2026-06-19 14:46:46 on dist-test-slave-cr22"
server_key: "d6fd83c34bdef520e3cc6cea62d47005"
server_key_iv: "dbc1251b26719ec255097f824d7b1da7"
server_key_version: "encryptionkey@0"
I20260619 14:46:46.327695 17672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:46.346112 17672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:46.349514 17672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:46.349753 17672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:46.350400 17672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:46.351320 17672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:46.351410 17672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:46.351519 17672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:46.351578 17672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:46.361654 17672 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41705
I20260619 14:46:46.361677 17800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41705 every 8 connection(s)
I20260619 14:46:46.362689 17672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:46.368485 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 17672
I20260619 14:46:46.368639 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:46:46.368971 392 external_mini_cluster.cc:1428] Setting key fcd7a9e961f4df0ac9e646c048fe5a2f
Jun 19 14:46:46 dist-test-slave-cr22 krb5kdc[17431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:46.379299 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.364733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55233 (local address 127.0.98.62:42789)
0619 14:46:46.364888 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:46.364893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:46.365863 (+ 970us) server_negotiation.cc:407] Connection header received
0619 14:46:46.367307 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:46.367310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:46.367362 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:46.367516 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:46.368979 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.369909 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.371538 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.371807 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.371934 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.374857 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:46.374864 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:46.374870 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:46.374905 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:46.376860 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:46.377423 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:46.377428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:46.377431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:46.377549 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:46.377901 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:46.377904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:46.377908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:46.378242 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:46.378355 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:46.378893 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 14:46:46.379046 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:46:46.380264 17803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.365161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42789 (local address 127.0.98.2:55233)
0619 14:46:46.365692 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:46.365743 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:46.367077 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:46.367709 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:46.367716 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:46.368153 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:46.368793 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.368812 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.370058 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.370062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.371358 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.371368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.371976 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.371981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.372173 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.372849 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:46.372868 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:46.374583 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:46.376995 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:46.377000 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:46.377013 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:46.377315 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:46.377655 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:46.377658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:46.377660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:46.377800 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:46.378366 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:46.378372 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:46.378616 (+ 244us) client_negotiation.cc:769] Sending connection context
0619 14:46:46.378804 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 14:46:46.379074 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":127,"threads_started":1}
I20260619 14:46:46.381733 17801 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42789
I20260619 14:46:46.382014 17801 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:46:46.382606 17801 heartbeater.cc:511] Master 127.0.98.62:42789 requested a full tablet report, sending...
I20260619 14:46:46.383934 17476 ts_manager.cc:194] Registered new tserver with Master: 89755e1bdae84acc8b22acb7bcbfa1a9 (127.0.98.2:41705)
I20260619 14:46:46.384717 17476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:46:46.402339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42789
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:46.518579 17806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:46.518879 17806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:46.518929 17806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:46.523627 17806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:46.523715 17806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:46.523782 17806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:46.529821 17806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:46.531003 17806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:46.532136 17806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:46.542085 17814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:46.542281 17806 server_base.cc:1034] running on GCE node
W20260619 14:46:46.542140 17811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:46.542207 17812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:46.542863 17806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:46.543619 17806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:46.544845 17806 hybrid_clock.cc:648] HybridClock initialized: now 1781880406544833 us; error 26 us; skew 500 ppm
Jun 19 14:46:46 dist-test-slave-cr22 krb5kdc[17431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:46.547933 17806 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:46:46.549082 17806 webserver.cc:460] Webserver started at http://127.0.98.3:43703/ using document root <none> and password file <none>
I20260619 14:46:46.549779 17806 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:46.549855 17806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:46.550089 17806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:46.552472 17806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "5958db4fd7134853a3eba51cd900e09a"
format_stamp: "Formatted at 2026-06-19 14:46:46 on dist-test-slave-cr22"
server_key: "a8407410867f4218def081ea1a427a41"
server_key_iv: "c5b9657676989a7512724f83fcbc89ea"
server_key_version: "encryptionkey@0"
I20260619 14:46:46.553120 17806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5958db4fd7134853a3eba51cd900e09a"
format_stamp: "Formatted at 2026-06-19 14:46:46 on dist-test-slave-cr22"
server_key: "a8407410867f4218def081ea1a427a41"
server_key_iv: "c5b9657676989a7512724f83fcbc89ea"
server_key_version: "encryptionkey@0"
I20260619 14:46:46.557381 17806 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:46:46.560205 17821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:46.561745 17806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:46:46.561892 17806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "5958db4fd7134853a3eba51cd900e09a"
format_stamp: "Formatted at 2026-06-19 14:46:46 on dist-test-slave-cr22"
server_key: "a8407410867f4218def081ea1a427a41"
server_key_iv: "c5b9657676989a7512724f83fcbc89ea"
server_key_version: "encryptionkey@0"
I20260619 14:46:46.562022 17806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:46.576781 17806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:46.580235 17806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:46.580448 17806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:46.581096 17806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:46.581950 17806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:46.582023 17806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:46.582104 17806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:46.582149 17806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:46.591537 17806 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32841
I20260619 14:46:46.591563 17934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32841 every 8 connection(s)
I20260619 14:46:46.592563 17806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:46.600010 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 17806
I20260619 14:46:46.600226 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:46:46.600548 392 external_mini_cluster.cc:1428] Setting key 826a5e3aac556832f4daabc03068506b
Jun 19 14:46:46 dist-test-slave-cr22 krb5kdc[17431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:46.606689 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.594500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54801 (local address 127.0.98.62:42789)
0619 14:46:46.594659 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:46.594663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:46.595653 (+ 990us) server_negotiation.cc:407] Connection header received
0619 14:46:46.596484 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:46.596487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:46.596532 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:46.596612 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:46.597752 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.598324 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.599530 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.599764 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.599856 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.603070 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:46.603073 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:46.603075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:46.603103 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:46.604535 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:46.605078 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:46.605081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:46.605083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:46.605172 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:46.605499 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:46.605502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:46.605503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:46.605712 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:46.605800 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:46.606431 (+ 631us) server_negotiation.cc:299] Negotiation successful
0619 14:46:46.606540 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:46:46.607542 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.594901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42789 (local address 127.0.98.3:54801)
0619 14:46:46.595377 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:46.595417 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:46.596287 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:46.596799 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:46.596806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:46.597149 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:46.597594 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.597607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.598447 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.598450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.599369 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.599378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.599917 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.599922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.600131 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.600857 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:46.600877 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:46.602923 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:46.604702 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:46.604707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:46.604719 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:46.604984 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:46.605260 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:46.605262 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:46.605264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:46.605391 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:46.605837 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:46.605843 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:46.606151 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 14:46:46.606336 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 14:46:46.606616 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":126,"threads_started":1}
I20260619 14:46:46.608847 17935 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42789
I20260619 14:46:46.609064 17935 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:46.609566 17935 heartbeater.cc:511] Master 127.0.98.62:42789 requested a full tablet report, sending...
I20260619 14:46:46.610666 17476 ts_manager.cc:194] Registered new tserver with Master: 5958db4fd7134853a3eba51cd900e09a (127.0.98.3:32841)
I20260619 14:46:46.611339 17476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54801
I20260619 14:46:46.614456 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:46:46.624039 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.616512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.0.98.62:42789)
0619 14:46:46.616690 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:46.616694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:46.616880 (+ 186us) server_negotiation.cc:407] Connection header received
0619 14:46:46.616965 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:46.616967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:46.617005 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:46.617068 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:46.617788 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.618287 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.619130 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.619341 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.619423 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.620892 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:46.620895 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:46.620897 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:46.620918 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:46.622245 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:46.622685 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:46.622688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:46.622689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:46.622760 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:46.623074 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:46.623077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:46.623078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:46.623301 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:46.623407 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:46.623744 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 14:46:46.623843 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 14:46:46.627293 17476 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:46:46.630476 17476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:46:46.645690 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.639851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42433 (local address 127.0.0.1:37424)
0619 14:46:46.640587 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:46.641201 (+ 614us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:46.641279 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:46.641929 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:46.641932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:46.641962 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:46.642103 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.642110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.643053 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.643060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.644320 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.644329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.645092 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.645096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.645205 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.645241 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:46:46.645377 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 14:46:46.645506 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":41,"spinlock_wait_cycles":1920,"thread_start_us":77,"threads_started":1}
I20260619 14:46:46.645697 17948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.640154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37424 (local address 127.0.98.1:42433)
0619 14:46:46.640823 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:46.640839 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:46.641273 (+ 434us) server_negotiation.cc:407] Connection header received
0619 14:46:46.641421 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:46.641426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:46.641603 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:46.641752 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:46.642229 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.642932 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.644443 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.644973 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.645093 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.645254 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:46.645405 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 14:46:46.645501 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":78,"threads_started":1}
I20260619 14:46:46.646128 17946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.639851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41705 (local address 127.0.0.1:44068)
0619 14:46:46.640587 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:46.640647 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:46.640768 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:46.641689 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:46.641695 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:46.641716 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:46.641905 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.641913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.643355 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.643359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.644629 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.644638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.645722 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.645726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.645840 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.645866 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:46:46.645910 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 14:46:46.645983 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":111,"thread_start_us":123,"threads_started":1}
I20260619 14:46:46.646183 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.640699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32841 (local address 127.0.0.1:34642)
0619 14:46:46.641002 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:46.641016 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:46.641111 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:46.641923 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:46.641925 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:46.641944 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:46.642075 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.642085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.643707 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.643711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.644805 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.644813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.645855 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.645859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.645941 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.645960 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:46:46.646000 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 14:46:46.646052 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":1,"spinlock_wait_cycles":1408,"thread_start_us":84,"threads_started":1}
I20260619 14:46:46.646780 17951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.641140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34642 (local address 127.0.98.3:32841)
0619 14:46:46.641476 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:46.641486 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:46.641505 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:46:46.641566 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:46.641571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:46.641739 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:46.641918 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:46.642289 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.643571 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.644937 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.645741 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.646309 (+ 568us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.646480 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:46.646530 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 14:46:46.646622 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":99,"threads_started":1}
I20260619 14:46:46.646832 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.640109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44068 (local address 127.0.98.2:41705)
0619 14:46:46.641041 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:46.641053 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:46.641080 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:46:46.641163 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:46.641168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:46.641400 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:46.641676 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:46.642044 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.643213 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.644775 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.645565 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.646434 (+ 869us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.646556 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:46.646609 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 14:46:46.646689 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":130,"threads_started":1}
I20260619 14:46:46.649324 17601 tablet_service.cc:1467] Processing CreateTablet for tablet fd6067ecdca142969c8339442c86f6cf (DEFAULT_TABLE table=test-table [id=4e7bfa4019544a55ba5aa6699d52fa55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:46.649708 17869 tablet_service.cc:1467] Processing CreateTablet for tablet fd6067ecdca142969c8339442c86f6cf (DEFAULT_TABLE table=test-table [id=4e7bfa4019544a55ba5aa6699d52fa55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:46.650403 17601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6067ecdca142969c8339442c86f6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:46.650097 17735 tablet_service.cc:1467] Processing CreateTablet for tablet fd6067ecdca142969c8339442c86f6cf (DEFAULT_TABLE table=test-table [id=4e7bfa4019544a55ba5aa6699d52fa55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:46.650776 17869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6067ecdca142969c8339442c86f6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:46.651019 17735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6067ecdca142969c8339442c86f6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:46.657132 17952 tablet_bootstrap.cc:492] T fd6067ecdca142969c8339442c86f6cf P 7fc19052081943108a860d2fda03b646: Bootstrap starting.
I20260619 14:46:46.658118 17953 tablet_bootstrap.cc:492] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a: Bootstrap starting.
I20260619 14:46:46.659292 17952 tablet_bootstrap.cc:654] T fd6067ecdca142969c8339442c86f6cf P 7fc19052081943108a860d2fda03b646: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:46.659790 17954 tablet_bootstrap.cc:492] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9: Bootstrap starting.
I20260619 14:46:46.659960 17953 tablet_bootstrap.cc:654] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:46.660235 17952 log.cc:824] T fd6067ecdca142969c8339442c86f6cf P 7fc19052081943108a860d2fda03b646: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:46.661109 17953 log.cc:824] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:46.662200 17954 tablet_bootstrap.cc:654] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:46.662417 17952 tablet_bootstrap.cc:492] T fd6067ecdca142969c8339442c86f6cf P 7fc19052081943108a860d2fda03b646: No bootstrap required, opened a new log
I20260619 14:46:46.662675 17952 ts_tablet_manager.cc:1397] T fd6067ecdca142969c8339442c86f6cf P 7fc19052081943108a860d2fda03b646: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:46:46.662873 17953 tablet_bootstrap.cc:492] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a: No bootstrap required, opened a new log
I20260619 14:46:46.663075 17954 log.cc:824] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:46.663120 17953 ts_tablet_manager.cc:1397] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:46:46.665151 17954 tablet_bootstrap.cc:492] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9: No bootstrap required, opened a new log
I20260619 14:46:46.665323 17954 ts_tablet_manager.cc:1397] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:46:46.666400 17953 raft_consensus.cc:348] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc19052081943108a860d2fda03b646" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42433 } } peers { permanent_uuid: "89755e1bdae84acc8b22acb7bcbfa1a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41705 } } peers { permanent_uuid: "5958db4fd7134853a3eba51cd900e09a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32841 } }
I20260619 14:46:46.666647 17953 raft_consensus.cc:374] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:46.666718 17953 raft_consensus.cc:729] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5958db4fd7134853a3eba51cd900e09a, State: Initialized, Role: FOLLOWER
I20260619 14:46:46.667198 17953 consensus_queue.cc:260] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc19052081943108a860d2fda03b646" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42433 } } peers { permanent_uuid: "89755e1bdae84acc8b22acb7bcbfa1a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41705 } } peers { permanent_uuid: "5958db4fd7134853a3eba51cd900e09a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32841 } }
I20260619 14:46:46.667135 17952 raft_consensus.cc:348] T fd6067ecdca142969c8339442c86f6cf P 7fc19052081943108a860d2fda03b646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc19052081943108a860d2fda03b646" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42433 } } peers { permanent_uuid: "89755e1bdae84acc8b22acb7bcbfa1a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41705 } } peers { permanent_uuid: "5958db4fd7134853a3eba51cd900e09a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32841 } }
I20260619 14:46:46.667368 17952 raft_consensus.cc:374] T fd6067ecdca142969c8339442c86f6cf P 7fc19052081943108a860d2fda03b646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:46.667536 17952 raft_consensus.cc:729] T fd6067ecdca142969c8339442c86f6cf P 7fc19052081943108a860d2fda03b646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fc19052081943108a860d2fda03b646, State: Initialized, Role: FOLLOWER
I20260619 14:46:46.668023 17952 consensus_queue.cc:260] T fd6067ecdca142969c8339442c86f6cf P 7fc19052081943108a860d2fda03b646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc19052081943108a860d2fda03b646" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42433 } } peers { permanent_uuid: "89755e1bdae84acc8b22acb7bcbfa1a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41705 } } peers { permanent_uuid: "5958db4fd7134853a3eba51cd900e09a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32841 } }
I20260619 14:46:46.668136 17953 ts_tablet_manager.cc:1428] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260619 14:46:46.668543 17935 heartbeater.cc:503] Master 127.0.98.62:42789 was elected leader, sending a full tablet report...
I20260619 14:46:46.668659 17667 heartbeater.cc:503] Master 127.0.98.62:42789 was elected leader, sending a full tablet report...
I20260619 14:46:46.669385 17954 raft_consensus.cc:348] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc19052081943108a860d2fda03b646" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42433 } } peers { permanent_uuid: "89755e1bdae84acc8b22acb7bcbfa1a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41705 } } peers { permanent_uuid: "5958db4fd7134853a3eba51cd900e09a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32841 } }
I20260619 14:46:46.669749 17954 raft_consensus.cc:374] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:46.669865 17954 raft_consensus.cc:729] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89755e1bdae84acc8b22acb7bcbfa1a9, State: Initialized, Role: FOLLOWER
I20260619 14:46:46.670459 17954 consensus_queue.cc:260] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc19052081943108a860d2fda03b646" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42433 } } peers { permanent_uuid: "89755e1bdae84acc8b22acb7bcbfa1a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41705 } } peers { permanent_uuid: "5958db4fd7134853a3eba51cd900e09a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32841 } }
I20260619 14:46:46.671147 17952 ts_tablet_manager.cc:1428] T fd6067ecdca142969c8339442c86f6cf P 7fc19052081943108a860d2fda03b646: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20260619 14:46:46.671588 17954 ts_tablet_manager.cc:1428] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260619 14:46:46.672009 17801 heartbeater.cc:503] Master 127.0.98.62:42789 was elected leader, sending a full tablet report...
I20260619 14:46:46.742527 17958 raft_consensus.cc:482] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:46.742779 17958 raft_consensus.cc:504] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc19052081943108a860d2fda03b646" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42433 } } peers { permanent_uuid: "89755e1bdae84acc8b22acb7bcbfa1a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41705 } } peers { permanent_uuid: "5958db4fd7134853a3eba51cd900e09a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32841 } }
I20260619 14:46:46.744081 17958 leader_election.cc:288] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fc19052081943108a860d2fda03b646 (127.0.98.1:42433), 89755e1bdae84acc8b22acb7bcbfa1a9 (127.0.98.2:41705)
I20260619 14:46:46.747923 17948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.744578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47135 (local address 127.0.98.1:42433)
0619 14:46:46.744739 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:46.744743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:46.744755 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:46:46.744799 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:46.744801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:46.744864 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:46.744942 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:46.745442 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.745941 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.746955 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.747361 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.747500 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.747570 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:46.747730 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 14:46:46.747800 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 14:46:46.747923 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.744347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42433 (local address 127.0.98.3:47135)
0619 14:46:46.744471 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:46.744485 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:46.744580 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:46.745179 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:46.745182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:46.745211 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:46.745321 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.745325 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.746048 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.746055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.746830 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.746838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.747490 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.747494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.747593 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.747621 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 14:46:46.747715 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 14:46:46.747786 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":3}
I20260619 14:46:46.748032 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.744492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43039 (local address 127.0.98.2:41705)
0619 14:46:46.744619 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:46.744626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:46.744869 (+ 243us) server_negotiation.cc:407] Connection header received
0619 14:46:46.744954 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:46.744957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:46.745001 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:46.745080 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:46.745441 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.745987 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.747003 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.747401 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.747593 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.747664 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:46.747845 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 14:46:46.747909 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:46:46.748112 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.744352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41705 (local address 127.0.98.3:43039)
0619 14:46:46.744773 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:46.744785 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:46.744876 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:46.745089 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:46.745091 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:46.745103 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:46.745324 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.745328 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.746099 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.746106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.746879 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.746887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.747596 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.747600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.747714 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.747733 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:46:46.747819 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 14:46:46.747896 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":65,"spinlock_wait_cycles":26368,"thread_start_us":113,"threads_started":1}
I20260619 14:46:46.748795 17621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6067ecdca142969c8339442c86f6cf" candidate_uuid: "5958db4fd7134853a3eba51cd900e09a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fc19052081943108a860d2fda03b646" is_pre_election: true
I20260619 14:46:46.748860 17755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6067ecdca142969c8339442c86f6cf" candidate_uuid: "5958db4fd7134853a3eba51cd900e09a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89755e1bdae84acc8b22acb7bcbfa1a9" is_pre_election: true
I20260619 14:46:46.749148 17755 raft_consensus.cc:2454] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5958db4fd7134853a3eba51cd900e09a in term 0.
I20260619 14:46:46.749111 17621 raft_consensus.cc:2454] T fd6067ecdca142969c8339442c86f6cf P 7fc19052081943108a860d2fda03b646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5958db4fd7134853a3eba51cd900e09a in term 0.
I20260619 14:46:46.749711 17822 leader_election.cc:302] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5958db4fd7134853a3eba51cd900e09a, 7fc19052081943108a860d2fda03b646, 89755e1bdae84acc8b22acb7bcbfa1a9; no voters:
I20260619 14:46:46.750007 17958 raft_consensus.cc:2781] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:46:46.750108 17958 raft_consensus.cc:482] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:46:46.750168 17958 raft_consensus.cc:3037] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:46.751197 17958 raft_consensus.cc:504] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc19052081943108a860d2fda03b646" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42433 } } peers { permanent_uuid: "89755e1bdae84acc8b22acb7bcbfa1a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41705 } } peers { permanent_uuid: "5958db4fd7134853a3eba51cd900e09a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32841 } }
I20260619 14:46:46.751631 17958 leader_election.cc:288] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [CANDIDATE]: Term 1 election: Requested vote from peers 7fc19052081943108a860d2fda03b646 (127.0.98.1:42433), 89755e1bdae84acc8b22acb7bcbfa1a9 (127.0.98.2:41705)
I20260619 14:46:46.752146 17755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6067ecdca142969c8339442c86f6cf" candidate_uuid: "5958db4fd7134853a3eba51cd900e09a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89755e1bdae84acc8b22acb7bcbfa1a9"
I20260619 14:46:46.752146 17621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd6067ecdca142969c8339442c86f6cf" candidate_uuid: "5958db4fd7134853a3eba51cd900e09a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fc19052081943108a860d2fda03b646"
I20260619 14:46:46.752277 17755 raft_consensus.cc:3037] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:46.752341 17621 raft_consensus.cc:3037] T fd6067ecdca142969c8339442c86f6cf P 7fc19052081943108a860d2fda03b646 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:46.753592 17621 raft_consensus.cc:2454] T fd6067ecdca142969c8339442c86f6cf P 7fc19052081943108a860d2fda03b646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5958db4fd7134853a3eba51cd900e09a in term 1.
I20260619 14:46:46.753592 17755 raft_consensus.cc:2454] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5958db4fd7134853a3eba51cd900e09a in term 1.
I20260619 14:46:46.753943 17823 leader_election.cc:302] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5958db4fd7134853a3eba51cd900e09a, 7fc19052081943108a860d2fda03b646; no voters:
I20260619 14:46:46.754213 17958 raft_consensus.cc:2781] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:46.754433 17958 raft_consensus.cc:686] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [term 1 LEADER]: Becoming Leader. State: Replica: 5958db4fd7134853a3eba51cd900e09a, State: Running, Role: LEADER
I20260619 14:46:46.754760 17958 consensus_queue.cc:237] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc19052081943108a860d2fda03b646" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42433 } } peers { permanent_uuid: "89755e1bdae84acc8b22acb7bcbfa1a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41705 } } peers { permanent_uuid: "5958db4fd7134853a3eba51cd900e09a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32841 } }
I20260619 14:46:46.757858 17476 catalog_manager.cc:5495] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5958db4fd7134853a3eba51cd900e09a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5958db4fd7134853a3eba51cd900e09a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc19052081943108a860d2fda03b646" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89755e1bdae84acc8b22acb7bcbfa1a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5958db4fd7134853a3eba51cd900e09a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32841 } health_report { overall_health: HEALTHY } } }
I20260619 14:46:46.770416 17951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.766497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34648 (local address 127.0.98.3:32841)
0619 14:46:46.766640 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:46.766643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:46.766654 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:46:46.766728 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:46.766730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:46.766790 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:46.766902 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:46.767153 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.767839 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.768780 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.769043 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.769150 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.769450 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:46.769988 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:46.770116 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:46.770186 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 14:46:46.770243 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:46:46.777168 17755 raft_consensus.cc:1261] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9 [term 1 FOLLOWER]: Refusing update from remote peer 5958db4fd7134853a3eba51cd900e09a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:46.777269 17621 raft_consensus.cc:1261] T fd6067ecdca142969c8339442c86f6cf P 7fc19052081943108a860d2fda03b646 [term 1 FOLLOWER]: Refusing update from remote peer 5958db4fd7134853a3eba51cd900e09a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:46.778016 17958 consensus_queue.cc:1035] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89755e1bdae84acc8b22acb7bcbfa1a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:46.778285 17962 consensus_queue.cc:1035] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fc19052081943108a860d2fda03b646" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:46.786950 17966 mvcc.cc:205] Tried to move back new op lower bound from 7298582146151260160 to 7298582146070577152. Current Snapshot: MvccSnapshot[applied={T|T < 7298582146151260160}]
I20260619 14:46:46.800150 17476 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:46.800343 17476 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:46.802627 17476 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e637c75b0354d8f8f210e11b1518fab: Sending DeleteTablet for 3 replicas of tablet fd6067ecdca142969c8339442c86f6cf
I20260619 14:46:46.803537 17735 tablet_service.cc:1514] Processing DeleteTablet for tablet fd6067ecdca142969c8339442c86f6cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44068
I20260619 14:46:46.803643 17869 tablet_service.cc:1514] Processing DeleteTablet for tablet fd6067ecdca142969c8339442c86f6cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34642
I20260619 14:46:46.804085 17975 tablet_replica.cc:330] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9: stopping tablet replica
I20260619 14:46:46.804067 17601 tablet_service.cc:1514] Processing DeleteTablet for tablet fd6067ecdca142969c8339442c86f6cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37424
I20260619 14:46:46.804392 17975 raft_consensus.cc:2229] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:46.804523 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 17538
I20260619 14:46:46.804693 17977 tablet_replica.cc:330] T fd6067ecdca142969c8339442c86f6cf P 7fc19052081943108a860d2fda03b646: stopping tablet replica
I20260619 14:46:46.804766 17975 raft_consensus.cc:2258] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:46.805184 17976 tablet_replica.cc:330] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a: stopping tablet replica
I20260619 14:46:46.805452 17976 raft_consensus.cc:2229] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:46:46.805838 17976 raft_consensus.cc:2258] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:46.807420 17976 ts_tablet_manager.cc:1905] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:46.807696 17975 ts_tablet_manager.cc:1905] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:46.810246 17976 ts_tablet_manager.cc:1918] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:46.810388 17976 log.cc:1196] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/fd6067ecdca142969c8339442c86f6cf
I20260619 14:46:46.810796 17976 ts_tablet_manager.cc:1939] T fd6067ecdca142969c8339442c86f6cf P 5958db4fd7134853a3eba51cd900e09a: Deleting consensus metadata
I20260619 14:46:46.810858 17975 ts_tablet_manager.cc:1918] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:46.810981 17975 log.cc:1196] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/fd6067ecdca142969c8339442c86f6cf
I20260619 14:46:46.811322 17975 ts_tablet_manager.cc:1939] T fd6067ecdca142969c8339442c86f6cf P 89755e1bdae84acc8b22acb7bcbfa1a9: Deleting consensus metadata
I20260619 14:46:46.812646 17460 catalog_manager.cc:4841] TS 89755e1bdae84acc8b22acb7bcbfa1a9 (127.0.98.2:41705): tablet fd6067ecdca142969c8339442c86f6cf (table test-table [id=4e7bfa4019544a55ba5aa6699d52fa55]) successfully deleted
W20260619 14:46:46.814335 17461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42433 (error 108)
I20260619 14:46:46.814879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 17672
I20260619 14:46:46.814900 17461 catalog_manager.cc:4841] TS 5958db4fd7134853a3eba51cd900e09a (127.0.98.3:32841): tablet fd6067ecdca142969c8339442c86f6cf (table test-table [id=4e7bfa4019544a55ba5aa6699d52fa55]) successfully deleted
I20260619 14:46:46.815670 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.815151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42433 (local address 127.0.0.1:37432)
0619 14:46:46.815519 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:46.815586 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265}
W20260619 14:46:46.815943 17461 catalog_manager.cc:4584] TS 7fc19052081943108a860d2fda03b646 (127.0.98.1:42433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd6067ecdca142969c8339442c86f6cf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42433: connect: Connection refused (error 111)
I20260619 14:46:46.821000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 17806
I20260619 14:46:46.826804 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 17444
2026-06-19T14:46:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2468 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T14:46:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:46Z Disabled control of system clock
I20260619 14:46:46.859485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33973
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:46:46.978765 17982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:46.979044 17982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:46.979082 17982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:46.983063 17982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:46.983127 17982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:46.983143 17982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:46.983166 17982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:46.983182 17982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:46.987917 17982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33973
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:46.988969 17982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:46.990295 17982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:46:46.999382 17982 server_base.cc:1034] running on GCE node
W20260619 14:46:46.999326 17988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:46.999323 17990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:46.999342 17987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:47.000317 17982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:47.001497 17982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:47.002725 17982 hybrid_clock.cc:648] HybridClock initialized: now 1781880407002704 us; error 33 us; skew 500 ppm
I20260619 14:46:47.008728 17982 webserver.cc:460] Webserver started at http://127.0.98.62:35991/ using document root <none> and password file <none>
I20260619 14:46:47.009485 17982 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:47.009598 17982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:47.009850 17982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:47.012089 17982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "6c2b0ec8a50a45a2a7507732e5f046f8"
format_stamp: "Formatted at 2026-06-19 14:46:47 on dist-test-slave-cr22"
server_key: "66baea2fee8e95fad0eee63951a743c7"
server_key_iv: "644800b1261646a96af5d8b2e88f4adc"
server_key_version: "encryptionkey@0"
I20260619 14:46:47.012692 17982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "6c2b0ec8a50a45a2a7507732e5f046f8"
format_stamp: "Formatted at 2026-06-19 14:46:47 on dist-test-slave-cr22"
server_key: "66baea2fee8e95fad0eee63951a743c7"
server_key_iv: "644800b1261646a96af5d8b2e88f4adc"
server_key_version: "encryptionkey@0"
I20260619 14:46:47.016721 17982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:46:47.019228 17996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:47.020637 17982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:46:47.020812 17982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "6c2b0ec8a50a45a2a7507732e5f046f8"
format_stamp: "Formatted at 2026-06-19 14:46:47 on dist-test-slave-cr22"
server_key: "66baea2fee8e95fad0eee63951a743c7"
server_key_iv: "644800b1261646a96af5d8b2e88f4adc"
server_key_version: "encryptionkey@0"
I20260619 14:46:47.020943 17982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:47.032294 17982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:47.033067 17982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:47.033293 17982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:47.041898 17982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33973
I20260619 14:46:47.041932 18048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33973 every 8 connection(s)
I20260619 14:46:47.042932 17982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:47.046286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 17982
I20260619 14:46:47.046460 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:47.046795 18049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:47.046797 392 external_mini_cluster.cc:1428] Setting key 4c90c005c4a4bfd0fac4cc137b8d69ed
I20260619 14:46:47.053614 18049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8: Bootstrap starting.
I20260619 14:46:47.056126 18049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:47.056902 18049 log.cc:824] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:47.057211 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.048583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.0.98.62:33973)
0619 14:46:47.049394 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:47.049418 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:47.049459 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:46:47.050378 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:47.050398 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:47.050808 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:47.051161 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:47.051648 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.052626 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.053508 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.054062 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.054232 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.055196 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:47.055200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:47.055214 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:47.055756 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:47.055824 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:47.055850 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:47.055965 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:47.056126 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 14:46:47.056427 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":137,"threads_started":1}
I20260619 14:46:47.059134 18049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8: No bootstrap required, opened a new log
I20260619 14:46:47.062856 18049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2b0ec8a50a45a2a7507732e5f046f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33973 } }
I20260619 14:46:47.063098 18049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:47.063177 18049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2b0ec8a50a45a2a7507732e5f046f8, State: Initialized, Role: FOLLOWER
I20260619 14:46:47.063652 18049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2b0ec8a50a45a2a7507732e5f046f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33973 } }
I20260619 14:46:47.063808 18049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:47.063896 18049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:47.064025 18049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:47.064946 18049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2b0ec8a50a45a2a7507732e5f046f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33973 } }
I20260619 14:46:47.065281 18049 leader_election.cc:302] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c2b0ec8a50a45a2a7507732e5f046f8; no voters:
I20260619 14:46:47.065598 18049 leader_election.cc:288] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:47.065848 18054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:47.066197 18054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8 [term 1 LEADER]: Becoming Leader. State: Replica: 6c2b0ec8a50a45a2a7507732e5f046f8, State: Running, Role: LEADER
I20260619 14:46:47.066540 18054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2b0ec8a50a45a2a7507732e5f046f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33973 } }
I20260619 14:46:47.066792 18049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:47.068274 18056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c2b0ec8a50a45a2a7507732e5f046f8. Latest consensus state: current_term: 1 leader_uuid: "6c2b0ec8a50a45a2a7507732e5f046f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2b0ec8a50a45a2a7507732e5f046f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33973 } } }
I20260619 14:46:47.068410 18056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:47.068363 18055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c2b0ec8a50a45a2a7507732e5f046f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2b0ec8a50a45a2a7507732e5f046f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33973 } } }
I20260619 14:46:47.068459 18055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8 [sys.catalog]: This master's current role is: LEADER
W20260619 14:46:47.071962 18069 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:46:47.072046 18069 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:46:47.072115 18065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:47.072944 18065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:47.079138 18065 catalog_manager.cc:1347] Generated new cluster ID: 2e4e66044f434ebc80e41082c5f0d2f1
I20260619 14:46:47.079304 18065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:47.096424 18065 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:47.097553 18065 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:47.108273 18065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8: Generated new TSK 0
I20260619 14:46:47.109091 18065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:46:47.112821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33973
--builtin_ntp_servers=127.0.98.20:34923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:46:47.228951 18073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:47.229236 18073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:47.229285 18073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:47.233430 18073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:47.233500 18073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:47.233557 18073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:47.238442 18073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:47.239678 18073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:47.241052 18073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:47.251150 18078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:47.251180 18079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:47.251189 18073 server_base.cc:1034] running on GCE node
W20260619 14:46:47.251147 18081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:47.251965 18073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:47.252712 18073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:47.253893 18073 hybrid_clock.cc:648] HybridClock initialized: now 1781880407253870 us; error 38 us; skew 500 ppm
I20260619 14:46:47.255908 18073 webserver.cc:460] Webserver started at http://127.0.98.1:45453/ using document root <none> and password file <none>
I20260619 14:46:47.256474 18073 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:47.256549 18073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:47.256717 18073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:47.258750 18073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "091fcf4be4074fbb9d9f6b1746dc84ea"
format_stamp: "Formatted at 2026-06-19 14:46:47 on dist-test-slave-cr22"
server_key: "94fa501f75d03921e7cc8e93ee82965d"
server_key_iv: "c90c092e2fb2d61f8ed925a49219d237"
server_key_version: "encryptionkey@0"
I20260619 14:46:47.259296 18073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "091fcf4be4074fbb9d9f6b1746dc84ea"
format_stamp: "Formatted at 2026-06-19 14:46:47 on dist-test-slave-cr22"
server_key: "94fa501f75d03921e7cc8e93ee82965d"
server_key_iv: "c90c092e2fb2d61f8ed925a49219d237"
server_key_version: "encryptionkey@0"
I20260619 14:46:47.263181 18073 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 14:46:47.266233 18087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:47.267941 18073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:47.268134 18073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "091fcf4be4074fbb9d9f6b1746dc84ea"
format_stamp: "Formatted at 2026-06-19 14:46:47 on dist-test-slave-cr22"
server_key: "94fa501f75d03921e7cc8e93ee82965d"
server_key_iv: "c90c092e2fb2d61f8ed925a49219d237"
server_key_version: "encryptionkey@0"
I20260619 14:46:47.268327 18073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:47.306488 18073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:47.307292 18073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:47.307595 18073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:47.308307 18073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:47.309348 18073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:47.309432 18073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:47.309508 18073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:47.309566 18073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:47.319011 18073 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34463
I20260619 14:46:47.319038 18200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34463 every 8 connection(s)
I20260619 14:46:47.320061 18073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:47.320564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 18073
I20260619 14:46:47.320712 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:46:47.321025 392 external_mini_cluster.cc:1428] Setting key bed07a355ffa130bcde6a4b9c4a8bc77
I20260619 14:46:47.323411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33973
--builtin_ntp_servers=127.0.98.20:34923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:46:47.332206 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.322101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60463 (local address 127.0.98.62:33973)
0619 14:46:47.322256 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:47.322262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:47.323782 (+ 1520us) server_negotiation.cc:407] Connection header received
0619 14:46:47.325578 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:47.325585 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:47.325656 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:47.325791 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:47.326860 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.327940 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.329263 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.329728 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.329880 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.331395 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:47.331401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:47.331406 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:47.331427 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:47.331496 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:47.331517 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:47.331617 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:47.331862 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 14:46:47.331950 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:46:47.333019 18203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.322750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33973 (local address 127.0.98.1:60463)
0619 14:46:47.323615 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:47.323681 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:47.325292 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:47.326022 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:47.326030 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:47.326124 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:47.326693 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:47.326708 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.328098 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.328102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:47.329117 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:47.329128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.329916 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.329919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:47.330074 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.331000 (+ 926us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:46:47.331002 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:47.331047 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:46:47.331198 (+ 151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:47.331676 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:47.331690 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:47.331696 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:46:47.331823 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 14:46:47.332068 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":185,"threads_started":1}
I20260619 14:46:47.334486 18201 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33973
I20260619 14:46:47.334779 18201 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:47.335350 18201 heartbeater.cc:511] Master 127.0.98.62:33973 requested a full tablet report, sending...
I20260619 14:46:47.336956 18010 ts_manager.cc:194] Registered new tserver with Master: 091fcf4be4074fbb9d9f6b1746dc84ea (127.0.98.1:34463)
I20260619 14:46:47.338299 18010 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:60463
W20260619 14:46:47.441646 18204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:47.441912 18204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:47.441951 18204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:47.446091 18204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:47.446156 18204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:47.446213 18204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:47.451252 18204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:47.452397 18204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:47.453780 18204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:47.463881 18212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:47.463881 18209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:47.463914 18210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:47.464185 18204 server_base.cc:1034] running on GCE node
I20260619 14:46:47.464884 18204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:47.465627 18204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:47.466928 18204 hybrid_clock.cc:648] HybridClock initialized: now 1781880407466909 us; error 34 us; skew 500 ppm
I20260619 14:46:47.469571 18204 webserver.cc:460] Webserver started at http://127.0.98.2:42603/ using document root <none> and password file <none>
I20260619 14:46:47.470225 18204 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:47.470289 18204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:47.470508 18204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:47.472812 18204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "e539c85e59264956b17ca3a2acd6dd02"
format_stamp: "Formatted at 2026-06-19 14:46:47 on dist-test-slave-cr22"
server_key: "c14622bb13266e4a09fbba72a7b96049"
server_key_iv: "64a9570cbdfcfc4731e5414ef5ae3e6a"
server_key_version: "encryptionkey@0"
I20260619 14:46:47.473425 18204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e539c85e59264956b17ca3a2acd6dd02"
format_stamp: "Formatted at 2026-06-19 14:46:47 on dist-test-slave-cr22"
server_key: "c14622bb13266e4a09fbba72a7b96049"
server_key_iv: "64a9570cbdfcfc4731e5414ef5ae3e6a"
server_key_version: "encryptionkey@0"
I20260619 14:46:47.477638 18204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:46:47.480518 18218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:47.481923 18204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:47.482072 18204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "e539c85e59264956b17ca3a2acd6dd02"
format_stamp: "Formatted at 2026-06-19 14:46:47 on dist-test-slave-cr22"
server_key: "c14622bb13266e4a09fbba72a7b96049"
server_key_iv: "64a9570cbdfcfc4731e5414ef5ae3e6a"
server_key_version: "encryptionkey@0"
I20260619 14:46:47.482210 18204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:47.502892 18204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:47.504122 18204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:47.504377 18204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:47.505059 18204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:47.506093 18204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:47.506166 18204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:47.506255 18204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:47.506385 18204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:47.518697 18204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34837
I20260619 14:46:47.518790 18331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34837 every 8 connection(s)
I20260619 14:46:47.519892 18204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:47.521266 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 18204
I20260619 14:46:47.521399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:46:47.521770 392 external_mini_cluster.cc:1428] Setting key eb6c0891390c446023d190588d934a63
I20260619 14:46:47.524696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33973
--builtin_ntp_servers=127.0.98.20:34923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:46:47.532137 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.522624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36855 (local address 127.0.98.62:33973)
0619 14:46:47.522779 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:47.522782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:47.525036 (+ 2254us) server_negotiation.cc:407] Connection header received
0619 14:46:47.526923 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:47.526926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:47.526989 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:47.527072 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:47.528112 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.528667 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.529837 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.530079 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.530190 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.531601 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:47.531605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:47.531607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:47.531621 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:47.531647 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:47.531656 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:47.531746 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:47.531915 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 14:46:47.531983 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:46:47.533033 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.523616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33973 (local address 127.0.98.2:36855)
0619 14:46:47.524881 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:47.524929 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:47.526678 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:47.527267 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:47.527274 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:47.527366 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:47.527958 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:47.527971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.528812 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.528817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:47.529705 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:47.529714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.530351 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.530363 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:47.530530 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.531258 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:46:47.531260 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:47.531306 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:46:47.531377 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:47.531748 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:47.531760 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:47.531767 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:46:47.531900 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 14:46:47.532213 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":405,"threads_started":1}
I20260619 14:46:47.534469 18332 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33973
I20260619 14:46:47.534848 18332 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:47.535431 18332 heartbeater.cc:511] Master 127.0.98.62:33973 requested a full tablet report, sending...
I20260619 14:46:47.536584 18010 ts_manager.cc:194] Registered new tserver with Master: e539c85e59264956b17ca3a2acd6dd02 (127.0.98.2:34837)
I20260619 14:46:47.537354 18010 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36855
W20260619 14:46:47.648849 18335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:47.649120 18335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:47.649453 18335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:47.653916 18335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:47.653980 18335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:47.654039 18335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:47.659046 18335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:47.660349 18335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:47.661737 18335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:47.671751 18343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:47.671751 18341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:47.671821 18340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:47.672060 18335 server_base.cc:1034] running on GCE node
I20260619 14:46:47.672685 18335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:47.673498 18335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:47.674775 18335 hybrid_clock.cc:648] HybridClock initialized: now 1781880407674739 us; error 54 us; skew 500 ppm
I20260619 14:46:47.676820 18335 webserver.cc:460] Webserver started at http://127.0.98.3:41297/ using document root <none> and password file <none>
I20260619 14:46:47.677408 18335 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:47.677503 18335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:47.677773 18335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:47.679914 18335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "e0b422e342fb4e768a22c95504d7ad6a"
format_stamp: "Formatted at 2026-06-19 14:46:47 on dist-test-slave-cr22"
server_key: "9c3881a0ec25b1bd79a5e744961023d6"
server_key_iv: "46b23a5f9f6849690813b7e49ed8dd01"
server_key_version: "encryptionkey@0"
I20260619 14:46:47.680466 18335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e0b422e342fb4e768a22c95504d7ad6a"
format_stamp: "Formatted at 2026-06-19 14:46:47 on dist-test-slave-cr22"
server_key: "9c3881a0ec25b1bd79a5e744961023d6"
server_key_iv: "46b23a5f9f6849690813b7e49ed8dd01"
server_key_version: "encryptionkey@0"
I20260619 14:46:47.684135 18335 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:46:47.686848 18349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:47.688282 18335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:47.688441 18335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "e0b422e342fb4e768a22c95504d7ad6a"
format_stamp: "Formatted at 2026-06-19 14:46:47 on dist-test-slave-cr22"
server_key: "9c3881a0ec25b1bd79a5e744961023d6"
server_key_iv: "46b23a5f9f6849690813b7e49ed8dd01"
server_key_version: "encryptionkey@0"
I20260619 14:46:47.688561 18335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:47.702265 18335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:47.702960 18335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:47.703163 18335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:47.703794 18335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:47.704759 18335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:47.704830 18335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:47.704900 18335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:47.704950 18335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:47.714337 18335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45303
I20260619 14:46:47.714359 18462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45303 every 8 connection(s)
I20260619 14:46:47.715252 18335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:47.722345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 18335
I20260619 14:46:47.722481 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:46:47.722779 392 external_mini_cluster.cc:1428] Setting key b612ab8ac60f9b97538fcd6ebc3a09fc
I20260619 14:46:47.726042 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.717126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40847 (local address 127.0.98.62:33973)
0619 14:46:47.717283 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:47.717287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:47.718414 (+ 1127us) server_negotiation.cc:407] Connection header received
0619 14:46:47.719873 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:47.719876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:47.719929 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:47.720016 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:47.721091 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.721764 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.723677 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.724019 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.724141 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.725548 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:47.725551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:47.725553 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:47.725566 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:47.725588 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:47.725596 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:47.725664 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:47.725866 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 14:46:47.725924 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:46:47.727191 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.717562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33973 (local address 127.0.98.3:40847)
0619 14:46:47.718243 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:47.718297 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:47.719643 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:47.720182 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:47.720188 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:47.720365 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:47.720910 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:47.720924 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.721928 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.721932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:47.723513 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:47.723528 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.724191 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.724196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:47.724475 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.725216 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:46:47.725217 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:47.725260 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:46:47.725327 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:47.725709 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:47.725718 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:47.725724 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:46:47.725851 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 14:46:47.726208 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":125,"threads_started":1}
I20260619 14:46:47.728536 18463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33973
I20260619 14:46:47.728901 18463 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:47.729502 18463 heartbeater.cc:511] Master 127.0.98.62:33973 requested a full tablet report, sending...
I20260619 14:46:47.730695 18010 ts_manager.cc:194] Registered new tserver with Master: e0b422e342fb4e768a22c95504d7ad6a (127.0.98.3:45303)
I20260619 14:46:47.731402 18010 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:40847
I20260619 14:46:47.737712 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:46:47.744179 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.740010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.0.98.62:33973)
0619 14:46:47.740215 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:47.740220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:47.740395 (+ 175us) server_negotiation.cc:407] Connection header received
0619 14:46:47.740560 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:47.740563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:47.740668 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:47.740764 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:47.741060 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.741596 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.742454 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.742686 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.742801 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.743701 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:47.743704 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:47.743706 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:47.743718 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:47.743736 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:47.743743 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:47.743827 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:47.743947 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 14:46:47.744006 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260619 14:46:47.747679 18010 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:46:47.750908 18010 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:46:47.767033 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.760540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45303 (local address 127.0.0.1:34750)
0619 14:46:47.761646 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:47.761705 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:47.761848 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:47.762417 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:47.762424 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:47.762447 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:47.762607 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:47.762613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.763963 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.763970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:47.765250 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:47.765260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.766241 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.766245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:47.766373 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.766418 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 14:46:47.766530 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 14:46:47.766855 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"thread_start_us":158,"threads_started":1}
I20260619 14:46:47.767074 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.761334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34750 (local address 127.0.98.3:45303)
0619 14:46:47.761852 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:47.761866 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:47.761896 (+ 30us) server_negotiation.cc:407] Connection header received
0619 14:46:47.761998 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:47.762003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:47.762248 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:47.762432 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:47.762737 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.763846 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.765387 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.766088 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.766245 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.766541 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:47.766597 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 14:46:47.766909 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":143,"threads_started":1}
I20260619 14:46:47.768697 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.760710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.0.98.1:34463)
0619 14:46:47.762541 (+ 1831us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:47.762552 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:47.762574 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:46:47.762648 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:47.762656 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:47.762822 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:47.762965 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:47.763223 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.764608 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.766352 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.767603 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.767717 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.768323 (+ 606us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:47.768394 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 14:46:47.768551 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1732,"thread_start_us":154,"threads_started":1}
I20260619 14:46:47.769066 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.760540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34463 (local address 127.0.0.1:49130)
0619 14:46:47.762071 (+ 1531us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:47.762084 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:47.762158 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:47.762965 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:47.762968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:47.762988 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:47.763088 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:47.763094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.764763 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.764767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:47.766192 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:47.766200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.768004 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.768007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:47.768099 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.768135 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:46:47.768197 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 14:46:47.768923 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1468,"mutex_wait_us":162,"thread_start_us":146,"threads_started":1}
I20260619 14:46:47.769443 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.760748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34837 (local address 127.0.0.1:60270)
0619 14:46:47.762162 (+ 1414us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:47.762174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:47.762289 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:47.762873 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:47.762879 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:47.762899 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:47.762997 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:47.763004 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.764830 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.764833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:47.766189 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:47.766200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.768930 (+ 2730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.768934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:47.769030 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.769052 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:46:47.769218 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 14:46:47.769319 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1365,"mutex_wait_us":111,"thread_start_us":111,"threads_started":1}
I20260619 14:46:47.770069 18479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.761448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.0.98.2:34837)
0619 14:46:47.762030 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:47.762039 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:47.762261 (+ 222us) server_negotiation.cc:407] Connection header received
0619 14:46:47.762398 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:47.762404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:47.762595 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:47.762734 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:47.763134 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.764638 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.766341 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.768674 (+ 2333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.768797 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.769094 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:47.769628 (+ 534us) server_negotiation.cc:299] Negotiation successful
0619 14:46:47.769905 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":134,"threads_started":1}
I20260619 14:46:47.770991 18397 tablet_service.cc:1467] Processing CreateTablet for tablet 0834d52d856645bb82dfd7df72c61034 (DEFAULT_TABLE table=test-table [id=922fed4573ca4a1f81f05924ebe7ccce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:47.772212 18397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0834d52d856645bb82dfd7df72c61034. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:47.772014 18135 tablet_service.cc:1467] Processing CreateTablet for tablet 0834d52d856645bb82dfd7df72c61034 (DEFAULT_TABLE table=test-table [id=922fed4573ca4a1f81f05924ebe7ccce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:47.773066 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0834d52d856645bb82dfd7df72c61034. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:47.772920 18264 tablet_service.cc:1467] Processing CreateTablet for tablet 0834d52d856645bb82dfd7df72c61034 (DEFAULT_TABLE table=test-table [id=922fed4573ca4a1f81f05924ebe7ccce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:47.773952 18264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0834d52d856645bb82dfd7df72c61034. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:47.778712 18480 tablet_bootstrap.cc:492] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a: Bootstrap starting.
I20260619 14:46:47.780858 18481 tablet_bootstrap.cc:492] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea: Bootstrap starting.
I20260619 14:46:47.781610 18480 tablet_bootstrap.cc:654] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:47.782395 18482 tablet_bootstrap.cc:492] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02: Bootstrap starting.
I20260619 14:46:47.782656 18480 log.cc:824] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:47.783135 18481 tablet_bootstrap.cc:654] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:47.784019 18481 log.cc:824] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:47.784559 18482 tablet_bootstrap.cc:654] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:47.785039 18480 tablet_bootstrap.cc:492] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a: No bootstrap required, opened a new log
I20260619 14:46:47.785241 18480 ts_tablet_manager.cc:1397] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 14:46:47.785346 18482 log.cc:824] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:47.785939 18481 tablet_bootstrap.cc:492] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea: No bootstrap required, opened a new log
I20260619 14:46:47.786124 18481 ts_tablet_manager.cc:1397] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:46:47.787226 18482 tablet_bootstrap.cc:492] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02: No bootstrap required, opened a new log
I20260619 14:46:47.787415 18482 ts_tablet_manager.cc:1397] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:46:47.790113 18481 raft_consensus.cc:348] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091fcf4be4074fbb9d9f6b1746dc84ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34463 } } peers { permanent_uuid: "e0b422e342fb4e768a22c95504d7ad6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45303 } } peers { permanent_uuid: "e539c85e59264956b17ca3a2acd6dd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34837 } }
I20260619 14:46:47.790061 18480 raft_consensus.cc:348] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091fcf4be4074fbb9d9f6b1746dc84ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34463 } } peers { permanent_uuid: "e0b422e342fb4e768a22c95504d7ad6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45303 } } peers { permanent_uuid: "e539c85e59264956b17ca3a2acd6dd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34837 } }
I20260619 14:46:47.790356 18480 raft_consensus.cc:374] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:47.790356 18481 raft_consensus.cc:374] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:47.790413 18480 raft_consensus.cc:729] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b422e342fb4e768a22c95504d7ad6a, State: Initialized, Role: FOLLOWER
I20260619 14:46:47.790413 18481 raft_consensus.cc:729] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 091fcf4be4074fbb9d9f6b1746dc84ea, State: Initialized, Role: FOLLOWER
I20260619 14:46:47.790884 18480 consensus_queue.cc:260] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091fcf4be4074fbb9d9f6b1746dc84ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34463 } } peers { permanent_uuid: "e0b422e342fb4e768a22c95504d7ad6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45303 } } peers { permanent_uuid: "e539c85e59264956b17ca3a2acd6dd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34837 } }
I20260619 14:46:47.790884 18481 consensus_queue.cc:260] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091fcf4be4074fbb9d9f6b1746dc84ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34463 } } peers { permanent_uuid: "e0b422e342fb4e768a22c95504d7ad6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45303 } } peers { permanent_uuid: "e539c85e59264956b17ca3a2acd6dd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34837 } }
I20260619 14:46:47.791025 18482 raft_consensus.cc:348] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091fcf4be4074fbb9d9f6b1746dc84ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34463 } } peers { permanent_uuid: "e0b422e342fb4e768a22c95504d7ad6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45303 } } peers { permanent_uuid: "e539c85e59264956b17ca3a2acd6dd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34837 } }
I20260619 14:46:47.791321 18482 raft_consensus.cc:374] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:47.791460 18482 raft_consensus.cc:729] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e539c85e59264956b17ca3a2acd6dd02, State: Initialized, Role: FOLLOWER
I20260619 14:46:47.791949 18480 ts_tablet_manager.cc:1428] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 14:46:47.791993 18201 heartbeater.cc:503] Master 127.0.98.62:33973 was elected leader, sending a full tablet report...
I20260619 14:46:47.791941 18482 consensus_queue.cc:260] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091fcf4be4074fbb9d9f6b1746dc84ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34463 } } peers { permanent_uuid: "e0b422e342fb4e768a22c95504d7ad6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45303 } } peers { permanent_uuid: "e539c85e59264956b17ca3a2acd6dd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34837 } }
I20260619 14:46:47.792682 18332 heartbeater.cc:503] Master 127.0.98.62:33973 was elected leader, sending a full tablet report...
I20260619 14:46:47.793053 18482 ts_tablet_manager.cc:1428] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:46:47.793350 18463 heartbeater.cc:503] Master 127.0.98.62:33973 was elected leader, sending a full tablet report...
I20260619 14:46:47.794456 18481 ts_tablet_manager.cc:1428] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260619 14:46:47.822206 18202 tablet.cc:2369] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:46:47.839533 18486 raft_consensus.cc:482] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:47.839797 18486 raft_consensus.cc:504] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091fcf4be4074fbb9d9f6b1746dc84ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34463 } } peers { permanent_uuid: "e0b422e342fb4e768a22c95504d7ad6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45303 } } peers { permanent_uuid: "e539c85e59264956b17ca3a2acd6dd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34837 } }
I20260619 14:46:47.840909 18486 leader_election.cc:288] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0b422e342fb4e768a22c95504d7ad6a (127.0.98.3:45303), e539c85e59264956b17ca3a2acd6dd02 (127.0.98.2:34837)
I20260619 14:46:47.846560 18479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.841381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52467 (local address 127.0.98.2:34837)
0619 14:46:47.841514 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:47.841519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:47.841800 (+ 281us) server_negotiation.cc:407] Connection header received
0619 14:46:47.841900 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:47.841903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:47.841959 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:47.842055 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:47.842453 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.843254 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.845168 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.845778 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.845909 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.845976 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:47.846340 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 14:46:47.846429 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:46:47.846560 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.841369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46649 (local address 127.0.98.3:45303)
0619 14:46:47.841512 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:47.841518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:47.841800 (+ 282us) server_negotiation.cc:407] Connection header received
0619 14:46:47.841902 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:47.841905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:47.841959 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:47.842055 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:47.842453 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.843253 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.845168 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.845778 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.845920 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.845976 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:47.846357 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 14:46:47.846430 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:46:47.846580 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.841191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45303 (local address 127.0.98.1:46649)
0619 14:46:47.841661 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:47.841678 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:47.841802 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:47.842078 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:47.842081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:47.842096 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:47.842255 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:47.842265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.843450 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.843485 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:47.845014 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:47.845026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.845953 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.845957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:47.846130 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.846245 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 14:46:47.846319 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:46:47.846393 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":206,"thread_start_us":125,"threads_started":1}
I20260619 14:46:47.846580 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.841191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34837 (local address 127.0.98.1:52467)
0619 14:46:47.841702 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:47.841716 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:47.841802 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:47.842078 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:47.842174 (+ 96us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:47.842187 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:47.842289 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:47.842298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.843466 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.843469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:47.845014 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:47.845026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.845953 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.845957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:47.846110 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.846180 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 14:46:47.846269 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 14:46:47.846375 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":187,"spinlock_wait_cycles":12544,"thread_start_us":98,"threads_started":1}
I20260619 14:46:47.847270 18417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0834d52d856645bb82dfd7df72c61034" candidate_uuid: "091fcf4be4074fbb9d9f6b1746dc84ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b422e342fb4e768a22c95504d7ad6a" is_pre_election: true
I20260619 14:46:47.847635 18417 raft_consensus.cc:2454] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 091fcf4be4074fbb9d9f6b1746dc84ea in term 0.
I20260619 14:46:47.847572 18286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0834d52d856645bb82dfd7df72c61034" candidate_uuid: "091fcf4be4074fbb9d9f6b1746dc84ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e539c85e59264956b17ca3a2acd6dd02" is_pre_election: true
I20260619 14:46:47.847951 18286 raft_consensus.cc:2454] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 091fcf4be4074fbb9d9f6b1746dc84ea in term 0.
I20260619 14:46:47.848205 18088 leader_election.cc:302] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 091fcf4be4074fbb9d9f6b1746dc84ea, e0b422e342fb4e768a22c95504d7ad6a; no voters:
I20260619 14:46:47.848552 18486 raft_consensus.cc:2781] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:46:47.848630 18486 raft_consensus.cc:482] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:46:47.848690 18486 raft_consensus.cc:3037] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:47.849885 18486 raft_consensus.cc:504] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091fcf4be4074fbb9d9f6b1746dc84ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34463 } } peers { permanent_uuid: "e0b422e342fb4e768a22c95504d7ad6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45303 } } peers { permanent_uuid: "e539c85e59264956b17ca3a2acd6dd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34837 } }
I20260619 14:46:47.850458 18486 leader_election.cc:288] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [CANDIDATE]: Term 1 election: Requested vote from peers e0b422e342fb4e768a22c95504d7ad6a (127.0.98.3:45303), e539c85e59264956b17ca3a2acd6dd02 (127.0.98.2:34837)
I20260619 14:46:47.850960 18286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0834d52d856645bb82dfd7df72c61034" candidate_uuid: "091fcf4be4074fbb9d9f6b1746dc84ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e539c85e59264956b17ca3a2acd6dd02"
I20260619 14:46:47.850981 18417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0834d52d856645bb82dfd7df72c61034" candidate_uuid: "091fcf4be4074fbb9d9f6b1746dc84ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0b422e342fb4e768a22c95504d7ad6a"
I20260619 14:46:47.851102 18286 raft_consensus.cc:3037] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:47.851102 18417 raft_consensus.cc:3037] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:47.852483 18286 raft_consensus.cc:2454] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 091fcf4be4074fbb9d9f6b1746dc84ea in term 1.
I20260619 14:46:47.852483 18417 raft_consensus.cc:2454] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 091fcf4be4074fbb9d9f6b1746dc84ea in term 1.
I20260619 14:46:47.852867 18088 leader_election.cc:302] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 091fcf4be4074fbb9d9f6b1746dc84ea, e0b422e342fb4e768a22c95504d7ad6a; no voters:
I20260619 14:46:47.853123 18486 raft_consensus.cc:2781] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:47.853387 18486 raft_consensus.cc:686] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [term 1 LEADER]: Becoming Leader. State: Replica: 091fcf4be4074fbb9d9f6b1746dc84ea, State: Running, Role: LEADER
I20260619 14:46:47.853760 18486 consensus_queue.cc:237] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091fcf4be4074fbb9d9f6b1746dc84ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34463 } } peers { permanent_uuid: "e0b422e342fb4e768a22c95504d7ad6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45303 } } peers { permanent_uuid: "e539c85e59264956b17ca3a2acd6dd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34837 } }
I20260619 14:46:47.857609 18009 catalog_manager.cc:5495] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 091fcf4be4074fbb9d9f6b1746dc84ea (127.0.98.1). New cstate: current_term: 1 leader_uuid: "091fcf4be4074fbb9d9f6b1746dc84ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091fcf4be4074fbb9d9f6b1746dc84ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0b422e342fb4e768a22c95504d7ad6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e539c85e59264956b17ca3a2acd6dd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34837 } health_report { overall_health: UNKNOWN } } }
I20260619 14:46:47.891994 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.888282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49134 (local address 127.0.98.1:34463)
0619 14:46:47.888445 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:47.888452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:47.888468 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:46:47.888556 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:47.888558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:47.888616 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:47.888740 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:47.888979 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.889496 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.890431 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.890656 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.890774 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.891089 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:47.891598 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:47.891735 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:47.891781 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 14:46:47.891838 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:46:47.898449 18417 raft_consensus.cc:1261] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a [term 1 FOLLOWER]: Refusing update from remote peer 091fcf4be4074fbb9d9f6b1746dc84ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:47.898449 18286 raft_consensus.cc:1261] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02 [term 1 FOLLOWER]: Refusing update from remote peer 091fcf4be4074fbb9d9f6b1746dc84ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:47.899220 18491 consensus_queue.cc:1035] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "e539c85e59264956b17ca3a2acd6dd02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:47.899403 18486 consensus_queue.cc:1035] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b422e342fb4e768a22c95504d7ad6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:47.909047 18496 mvcc.cc:205] Tried to move back new op lower bound from 7298582150744141824 to 7298582150571819008. Current Snapshot: MvccSnapshot[applied={T|T < 7298582150744141824}]
I20260619 14:46:47.916754 18010 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:47.916939 18010 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:47.919340 18010 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c2b0ec8a50a45a2a7507732e5f046f8: Sending DeleteTablet for 3 replicas of tablet 0834d52d856645bb82dfd7df72c61034
I20260619 14:46:47.920202 18397 tablet_service.cc:1514] Processing DeleteTablet for tablet 0834d52d856645bb82dfd7df72c61034 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:47 UTC) from {username='slave'} at 127.0.0.1:34750
I20260619 14:46:47.920295 18135 tablet_service.cc:1514] Processing DeleteTablet for tablet 0834d52d856645bb82dfd7df72c61034 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:47 UTC) from {username='slave'} at 127.0.0.1:49130
I20260619 14:46:47.920377 18264 tablet_service.cc:1514] Processing DeleteTablet for tablet 0834d52d856645bb82dfd7df72c61034 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:47 UTC) from {username='slave'} at 127.0.0.1:60270
I20260619 14:46:47.920850 18505 tablet_replica.cc:330] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea: stopping tablet replica
I20260619 14:46:47.920964 18506 tablet_replica.cc:330] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02: stopping tablet replica
I20260619 14:46:47.921283 18505 raft_consensus.cc:2229] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:46:47.921293 18506 raft_consensus.cc:2229] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:47.921576 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 18073
I20260619 14:46:47.921629 18506 raft_consensus.cc:2258] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:47.921888 18505 raft_consensus.cc:2258] T 0834d52d856645bb82dfd7df72c61034 P 091fcf4be4074fbb9d9f6b1746dc84ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:47.922272 18504 tablet_replica.cc:330] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a: stopping tablet replica
I20260619 14:46:47.922626 18504 raft_consensus.cc:2229] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:47.922957 18504 raft_consensus.cc:2258] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:47.923987 18506 ts_tablet_manager.cc:1905] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:47.924540 18504 ts_tablet_manager.cc:1905] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:47.927838 18506 ts_tablet_manager.cc:1918] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:47.927968 18506 log.cc:1196] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/0834d52d856645bb82dfd7df72c61034
I20260619 14:46:47.928242 18504 ts_tablet_manager.cc:1918] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:47.928329 18506 ts_tablet_manager.cc:1939] T 0834d52d856645bb82dfd7df72c61034 P e539c85e59264956b17ca3a2acd6dd02: Deleting consensus metadata
I20260619 14:46:47.928369 18504 log.cc:1196] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/0834d52d856645bb82dfd7df72c61034
I20260619 14:46:47.928668 18504 ts_tablet_manager.cc:1939] T 0834d52d856645bb82dfd7df72c61034 P e0b422e342fb4e768a22c95504d7ad6a: Deleting consensus metadata
I20260619 14:46:47.929826 17997 catalog_manager.cc:4841] TS e0b422e342fb4e768a22c95504d7ad6a (127.0.98.3:45303): tablet 0834d52d856645bb82dfd7df72c61034 (table test-table [id=922fed4573ca4a1f81f05924ebe7ccce]) successfully deleted
I20260619 14:46:47.929833 17999 catalog_manager.cc:4841] TS e539c85e59264956b17ca3a2acd6dd02 (127.0.98.2:34837): tablet 0834d52d856645bb82dfd7df72c61034 (table test-table [id=922fed4573ca4a1f81f05924ebe7ccce]) successfully deleted
W20260619 14:46:47.930806 18000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34463 (error 108)
I20260619 14:46:47.930841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 18204
I20260619 14:46:47.932524 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.932244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34463 (local address 127.0.0.1:49150)
0619 14:46:47.932378 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:47.932450 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 14:46:47.932715 18000 catalog_manager.cc:4584] TS 091fcf4be4074fbb9d9f6b1746dc84ea (127.0.98.1:34463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0834d52d856645bb82dfd7df72c61034: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34463: connect: Connection refused (error 111)
I20260619 14:46:47.937158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 18335
I20260619 14:46:47.943303 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 17982
2026-06-19T14:46:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1115 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:46:48 dist-test-slave-cr22 krb5kdc[18510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:46:48 dist-test-slave-cr22 krb5kdc[18510](info): set up 2 sockets
Jun 19 14:46:48 dist-test-slave-cr22 krb5kdc[18510](info): commencing operation
krb5kdc: starting...
W20260619 14:46:49.015513 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:46:49 dist-test-slave-cr22 krb5kdc[18510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:46:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:46:49.147069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42493
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:49.259088 18523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:49.259377 18523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:49.259419 18523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:49.263577 18523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:49.263643 18523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:49.263664 18523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:49.263680 18523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:49.263696 18523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:49.269098 18523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:49.270270 18523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:49.271311 18523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:49.281049 18531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:49.281031 18529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:49.281085 18523 server_base.cc:1034] running on GCE node
W20260619 14:46:49.281028 18528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:49.281998 18523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:49.283324 18523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:49.284528 18523 hybrid_clock.cc:648] HybridClock initialized: now 1781880409284508 us; error 38 us; skew 500 ppm
Jun 19 14:46:49 dist-test-slave-cr22 krb5kdc[18510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:49.287669 18523 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:46:49.288614 18523 webserver.cc:460] Webserver started at http://127.0.98.62:40759/ using document root <none> and password file <none>
I20260619 14:46:49.289309 18523 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:49.289409 18523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:49.289650 18523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:49.291857 18523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "2365703ec7424c6d9f84c47a762e701a"
format_stamp: "Formatted at 2026-06-19 14:46:49 on dist-test-slave-cr22"
server_key: "23fc7aebed0fbfea88b834cb9d42e176"
server_key_iv: "01ab68fc3e77ffaa65743a1faaac7bb7"
server_key_version: "encryptionkey@0"
I20260619 14:46:49.292477 18523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "2365703ec7424c6d9f84c47a762e701a"
format_stamp: "Formatted at 2026-06-19 14:46:49 on dist-test-slave-cr22"
server_key: "23fc7aebed0fbfea88b834cb9d42e176"
server_key_iv: "01ab68fc3e77ffaa65743a1faaac7bb7"
server_key_version: "encryptionkey@0"
I20260619 14:46:49.296619 18523 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:46:49.299167 18538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:49.300760 18523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:49.300936 18523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "2365703ec7424c6d9f84c47a762e701a"
format_stamp: "Formatted at 2026-06-19 14:46:49 on dist-test-slave-cr22"
server_key: "23fc7aebed0fbfea88b834cb9d42e176"
server_key_iv: "01ab68fc3e77ffaa65743a1faaac7bb7"
server_key_version: "encryptionkey@0"
I20260619 14:46:49.301083 18523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:49.315366 18523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:49.318516 18523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:49.318758 18523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:49.325922 18523 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42493
I20260619 14:46:49.325953 18590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42493 every 8 connection(s)
I20260619 14:46:49.326869 18523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:49.329623 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:49.333878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 18523
I20260619 14:46:49.333988 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:49.334280 392 external_mini_cluster.cc:1428] Setting key 09d650c1c72595c0a2921ee1b768cb5c
I20260619 14:46:49.335651 18591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a: Bootstrap starting.
I20260619 14:46:49.338460 18591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:49.339399 18591 log.cc:824] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:49.341593 18591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a: No bootstrap required, opened a new log
Jun 19 14:46:49 dist-test-slave-cr22 krb5kdc[18510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:49.346124 18591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2365703ec7424c6d9f84c47a762e701a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42493 } }
I20260619 14:46:49.346416 18591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:49.346467 18591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2365703ec7424c6d9f84c47a762e701a, State: Initialized, Role: FOLLOWER
I20260619 14:46:49.346963 18591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2365703ec7424c6d9f84c47a762e701a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42493 } }
I20260619 14:46:49.347152 18591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:49.347237 18591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:49.347347 18591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:49.348551 18591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2365703ec7424c6d9f84c47a762e701a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42493 } }
I20260619 14:46:49.348912 18591 leader_election.cc:302] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2365703ec7424c6d9f84c47a762e701a; no voters:
I20260619 14:46:49.349314 18591 leader_election.cc:288] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:49.349488 18596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:49.349740 18596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a [term 1 LEADER]: Becoming Leader. State: Replica: 2365703ec7424c6d9f84c47a762e701a, State: Running, Role: LEADER
I20260619 14:46:49.350057 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:49.335786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47566 (local address 127.0.98.62:42493)
0619 14:46:49.336244 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:49.336267 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:49.336308 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:46:49.337183 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:49.337206 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:49.337614 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:49.338027 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:49.338839 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.340183 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.341111 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.341786 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.341958 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:49.344720 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:49.344726 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:49.344745 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:49.344776 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:49.346881 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:49.347334 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:49.347340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:49.347347 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:49.347517 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:49.347836 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:49.347839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:49.347841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:49.348352 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:49.348553 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:49.348774 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 14:46:49.349081 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":119,"threads_started":1}
I20260619 14:46:49.350099 18596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2365703ec7424c6d9f84c47a762e701a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42493 } }
I20260619 14:46:49.350637 18591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:49.351789 18598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2365703ec7424c6d9f84c47a762e701a. Latest consensus state: current_term: 1 leader_uuid: "2365703ec7424c6d9f84c47a762e701a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2365703ec7424c6d9f84c47a762e701a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42493 } } }
I20260619 14:46:49.351912 18598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:49.352334 18597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2365703ec7424c6d9f84c47a762e701a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2365703ec7424c6d9f84c47a762e701a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42493 } } }
I20260619 14:46:49.352435 18597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a [sys.catalog]: This master's current role is: LEADER
W20260619 14:46:49.355809 18611 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:46:49.355923 18611 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:46:49.356024 18605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:49.356756 18605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:49.363376 18605 catalog_manager.cc:1347] Generated new cluster ID: d7a3bc7b061e45bd8841a0831a06d2c9
I20260619 14:46:49.363528 18605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:49.378293 18605 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:49.379681 18605 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:49.384716 18605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a: Generated new TSK 0
I20260619 14:46:49.385468 18605 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:46:49.428131 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42493
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:49.583539 18617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:49.583830 18617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:49.583877 18617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:49.588488 18617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:49.588569 18617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:49.588672 18617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:49.594775 18617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:49.596140 18617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:49.597273 18617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:49.608274 18622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:49.608287 18623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:49.608414 18625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:49.608865 18617 server_base.cc:1034] running on GCE node
I20260619 14:46:49.609274 18617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:49.609975 18617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:49.611232 18617 hybrid_clock.cc:648] HybridClock initialized: now 1781880409611213 us; error 75 us; skew 500 ppm
Jun 19 14:46:49 dist-test-slave-cr22 krb5kdc[18510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:49.614372 18617 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:46:49.615296 18617 webserver.cc:460] Webserver started at http://127.0.98.1:33407/ using document root <none> and password file <none>
I20260619 14:46:49.615990 18617 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:49.616089 18617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:49.616330 18617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:49.618391 18617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "6ed99984067149ca8dc1c845245e516f"
format_stamp: "Formatted at 2026-06-19 14:46:49 on dist-test-slave-cr22"
server_key: "14b3299bf13a2d094ddc3d25d715dbe3"
server_key_iv: "5fb4762c8edd2c12f7ae8bf662a71eea"
server_key_version: "encryptionkey@0"
I20260619 14:46:49.618956 18617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6ed99984067149ca8dc1c845245e516f"
format_stamp: "Formatted at 2026-06-19 14:46:49 on dist-test-slave-cr22"
server_key: "14b3299bf13a2d094ddc3d25d715dbe3"
server_key_iv: "5fb4762c8edd2c12f7ae8bf662a71eea"
server_key_version: "encryptionkey@0"
I20260619 14:46:49.622699 18617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:46:49.625258 18632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:49.626549 18617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:46:49.626749 18617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "6ed99984067149ca8dc1c845245e516f"
format_stamp: "Formatted at 2026-06-19 14:46:49 on dist-test-slave-cr22"
server_key: "14b3299bf13a2d094ddc3d25d715dbe3"
server_key_iv: "5fb4762c8edd2c12f7ae8bf662a71eea"
server_key_version: "encryptionkey@0"
I20260619 14:46:49.626874 18617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:49.658816 18617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:49.662124 18617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:49.662360 18617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:49.662988 18617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:49.663961 18617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:49.664032 18617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:49.664108 18617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:49.664155 18617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:49.673472 18617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45175
I20260619 14:46:49.673496 18745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45175 every 8 connection(s)
I20260619 14:46:49.674414 18617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:49.677044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 18617
I20260619 14:46:49.677174 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:46:49.677510 392 external_mini_cluster.cc:1428] Setting key 3e9903b1db10072367f6170ffd3ff1c9
Jun 19 14:46:49 dist-test-slave-cr22 krb5kdc[18510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:46:49.690330 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:49.676679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45943 (local address 127.0.98.62:42493)
0619 14:46:49.676914 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:49.676922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:49.677875 (+ 953us) server_negotiation.cc:407] Connection header received
0619 14:46:49.679532 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:49.679534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:49.679579 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:49.679649 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:49.681249 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.681788 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.683011 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.683245 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.683307 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:49.686033 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:49.686044 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:49.686047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:49.686081 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:49.687851 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:49.688380 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:49.688384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:49.688386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:49.688473 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:49.688917 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:49.688921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:49.688923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:49.689119 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:49.689237 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:49.689912 (+ 675us) server_negotiation.cc:299] Negotiation successful
0619 14:46:49.690058 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 14:46:49.691005 18748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:49.677086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42493 (local address 127.0.98.1:45943)
0619 14:46:49.677723 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:49.677768 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:49.679233 (+ 1465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:49.679832 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:49.679845 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:49.680398 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:49.681098 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:49.681113 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.681934 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.681937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:49.682823 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:49.682833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.683534 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.683544 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:49.683698 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:49.684346 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:49.684363 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:49.685858 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:49.687978 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:49.687985 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:49.688000 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:49.688269 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:49.688568 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:49.688572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:49.688574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:49.688789 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:49.689235 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:49.689242 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:49.689590 (+ 348us) client_negotiation.cc:769] Sending connection context
0619 14:46:49.689822 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 14:46:49.690104 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":136,"threads_started":1}
I20260619 14:46:49.692792 18746 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42493
I20260619 14:46:49.693073 18746 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:49.693643 18746 heartbeater.cc:511] Master 127.0.98.62:42493 requested a full tablet report, sending...
I20260619 14:46:49.695505 18555 ts_manager.cc:194] Registered new tserver with Master: 6ed99984067149ca8dc1c845245e516f (127.0.98.1:45175)
I20260619 14:46:49.697163 18555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:46:49.707003 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42493
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:49.818820 18751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:49.819065 18751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:49.819146 18751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:49.823199 18751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:49.823257 18751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:49.823359 18751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:49.829008 18751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:49.830160 18751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:49.831265 18751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:49.841058 18759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:49.841075 18751 server_base.cc:1034] running on GCE node
W20260619 14:46:49.841114 18756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:49.841069 18757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:49.841758 18751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:49.842516 18751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:49.843685 18751 hybrid_clock.cc:648] HybridClock initialized: now 1781880409843654 us; error 44 us; skew 500 ppm
Jun 19 14:46:49 dist-test-slave-cr22 krb5kdc[18510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:49.846494 18751 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:46:49.847339 18751 webserver.cc:460] Webserver started at http://127.0.98.2:37149/ using document root <none> and password file <none>
I20260619 14:46:49.847919 18751 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:49.847980 18751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:49.848207 18751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:49.850239 18751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "b222360b7fec4ad8919bcbb507d3a428"
format_stamp: "Formatted at 2026-06-19 14:46:49 on dist-test-slave-cr22"
server_key: "157350b8d9350692e86c6ffc38aa0113"
server_key_iv: "078eb3688a88d8e51edc202b862b0270"
server_key_version: "encryptionkey@0"
I20260619 14:46:49.850891 18751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b222360b7fec4ad8919bcbb507d3a428"
format_stamp: "Formatted at 2026-06-19 14:46:49 on dist-test-slave-cr22"
server_key: "157350b8d9350692e86c6ffc38aa0113"
server_key_iv: "078eb3688a88d8e51edc202b862b0270"
server_key_version: "encryptionkey@0"
I20260619 14:46:49.854904 18751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:46:49.857421 18766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:49.858692 18751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:46:49.858860 18751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "b222360b7fec4ad8919bcbb507d3a428"
format_stamp: "Formatted at 2026-06-19 14:46:49 on dist-test-slave-cr22"
server_key: "157350b8d9350692e86c6ffc38aa0113"
server_key_iv: "078eb3688a88d8e51edc202b862b0270"
server_key_version: "encryptionkey@0"
I20260619 14:46:49.858984 18751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:49.874145 18751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:49.877458 18751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:49.877661 18751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:49.878285 18751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:49.879107 18751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:49.879174 18751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:49.879242 18751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:49.879285 18751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:49.888362 18751 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43567
I20260619 14:46:49.888379 18879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43567 every 8 connection(s)
I20260619 14:46:49.889249 18751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:49.893613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 18751
I20260619 14:46:49.893739 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:46:49.893987 392 external_mini_cluster.cc:1428] Setting key 3f597a92f31f2cb8c24645d612802b39
Jun 19 14:46:49 dist-test-slave-cr22 krb5kdc[18510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:46:49.905176 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:49.891002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33495 (local address 127.0.98.62:42493)
0619 14:46:49.891141 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:49.891144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:49.892133 (+ 989us) server_negotiation.cc:407] Connection header received
0619 14:46:49.894359 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:49.894362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:49.894411 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:49.894515 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:49.895959 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.896497 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.897762 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.897988 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.898102 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:49.901040 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:49.901043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:49.901045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:49.901069 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:49.902838 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:49.903414 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:49.903424 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:49.903426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:49.903572 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:49.903944 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:49.903947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:49.903949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:49.904144 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:49.904252 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:49.904877 (+ 625us) server_negotiation.cc:299] Negotiation successful
0619 14:46:49.905006 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:46:49.906174 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:49.891428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42493 (local address 127.0.98.2:33495)
0619 14:46:49.891968 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:49.892012 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:49.893201 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:49.894704 (+ 1503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:49.894711 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:49.895183 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:49.895789 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:49.895802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.896641 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.896645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:49.897630 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:49.897640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.898155 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.898159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:49.898324 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:49.898949 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:49.898965 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:49.900852 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:49.902991 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:49.902998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:49.903015 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:49.903295 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:49.903674 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:49.903677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:49.903679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:49.903839 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:49.904263 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:49.904272 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:49.904570 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 14:46:49.904820 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 14:46:49.905168 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:46:49.907745 18880 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42493
I20260619 14:46:49.908027 18880 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:49.908550 18880 heartbeater.cc:511] Master 127.0.98.62:42493 requested a full tablet report, sending...
I20260619 14:46:49.909680 18555 ts_manager.cc:194] Registered new tserver with Master: b222360b7fec4ad8919bcbb507d3a428 (127.0.98.2:43567)
I20260619 14:46:49.910318 18555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:46:49.923879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42493
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:50.039801 18885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:50.040061 18885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:50.040109 18885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:50.044598 18885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:50.044804 18885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:50.044957 18885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:50.053978 18885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:50.055251 18885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:50.056459 18885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:50.067880 18891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:50.067943 18893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:50.067895 18890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:50.068171 18885 server_base.cc:1034] running on GCE node
I20260619 14:46:50.068810 18885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:50.069667 18885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:50.070884 18885 hybrid_clock.cc:648] HybridClock initialized: now 1781880410070852 us; error 45 us; skew 500 ppm
Jun 19 14:46:50 dist-test-slave-cr22 krb5kdc[18510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:50.073812 18885 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:46:50.074717 18885 webserver.cc:460] Webserver started at http://127.0.98.3:32923/ using document root <none> and password file <none>
I20260619 14:46:50.075359 18885 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:50.075480 18885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:50.075718 18885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:50.078111 18885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "aa5cb4f3994445059b3b453cd3048ae6"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-cr22"
server_key: "cae63ec467a4f3500ffb7ea900c45824"
server_key_iv: "661d9bde0d75e13b2a73c396a19c3a0c"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.078822 18885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aa5cb4f3994445059b3b453cd3048ae6"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-cr22"
server_key: "cae63ec467a4f3500ffb7ea900c45824"
server_key_iv: "661d9bde0d75e13b2a73c396a19c3a0c"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.083025 18885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260619 14:46:50.085903 18900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:50.087361 18885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:46:50.087548 18885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "aa5cb4f3994445059b3b453cd3048ae6"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-cr22"
server_key: "cae63ec467a4f3500ffb7ea900c45824"
server_key_iv: "661d9bde0d75e13b2a73c396a19c3a0c"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.087682 18885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:50.103953 18885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:50.107735 18885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:50.107956 18885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:50.108570 18885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:50.109436 18885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:50.109509 18885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:50.109581 18885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:50.109642 18885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:50.118826 18885 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45471
I20260619 14:46:50.118853 19013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45471 every 8 connection(s)
I20260619 14:46:50.119751 18885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:50.121704 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 18885
I20260619 14:46:50.121850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:46:50.122123 392 external_mini_cluster.cc:1428] Setting key e0cc14ee4d8ed97a25d154832aee720e
Jun 19 14:46:50 dist-test-slave-cr22 krb5kdc[18510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:50.134562 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.121767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39475 (local address 127.0.98.62:42493)
0619 14:46:50.121924 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:50.121929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:50.122801 (+ 872us) server_negotiation.cc:407] Connection header received
0619 14:46:50.124162 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:50.124167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:50.124301 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:50.124373 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:50.125683 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.126201 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.127392 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.127705 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.127797 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.130516 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:50.130524 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:50.130526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:50.130550 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:50.132402 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:50.132940 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:50.132944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:50.132946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:50.133033 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:50.133448 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:50.133451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:50.133453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:50.133656 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:50.133770 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:50.134270 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 14:46:50.134394 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:46:50.135805 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.122159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42493 (local address 127.0.98.3:39475)
0619 14:46:50.122657 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:50.122709 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:50.123980 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:50.124605 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:50.124647 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:50.125069 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:50.125534 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.125547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.126341 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.126345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.127250 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.127260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.127857 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.127860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.128007 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.128654 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:50.128669 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:50.130311 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:50.132546 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:50.132552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:50.132567 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:50.132833 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:50.133197 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:50.133200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:50.133202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:50.133350 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:50.133767 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:50.133774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:50.134029 (+ 255us) client_negotiation.cc:769] Sending connection context
0619 14:46:50.134219 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 14:46:50.134564 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":123,"threads_started":1}
I20260619 14:46:50.137241 19014 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42493
I20260619 14:46:50.137465 19014 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:50.137940 19014 heartbeater.cc:511] Master 127.0.98.62:42493 requested a full tablet report, sending...
I20260619 14:46:50.138988 18555 ts_manager.cc:194] Registered new tserver with Master: aa5cb4f3994445059b3b453cd3048ae6 (127.0.98.3:45471)
I20260619 14:46:50.139624 18555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39475
I20260619 14:46:50.149036 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:46:50.159951 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.151342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47580 (local address 127.0.98.62:42493)
0619 14:46:50.151533 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:50.151538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:50.151784 (+ 246us) server_negotiation.cc:407] Connection header received
0619 14:46:50.151910 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:50.151913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:50.151958 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:50.152027 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:50.152862 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.153391 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.154239 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.154446 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.154525 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.156363 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:50.156366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:50.156368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:50.156392 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:50.157942 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:50.158482 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:50.158485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:50.158486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:50.158567 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:50.158909 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:50.158911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:50.158913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:50.159121 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:50.159248 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:50.159594 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 14:46:50.159753 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 14:46:50.163225 18555 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:46:50.166384 18555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:46:50.182174 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.175362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43567 (local address 127.0.0.1:54102)
0619 14:46:50.176463 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:50.176518 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:50.176647 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:50.177544 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:50.177550 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:50.177603 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:50.177800 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.177807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.179421 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.179425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.180764 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.180773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.181611 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.181615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.181721 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.181760 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 14:46:50.181872 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 14:46:50.182001 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":90,"spinlock_wait_cycles":10880,"thread_start_us":174,"threads_started":1}
I20260619 14:46:50.182370 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.176284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.0.98.1:45175)
0619 14:46:50.177035 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:50.177061 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:50.177080 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:46:50.177152 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:50.177158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:50.177364 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:50.177524 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:50.177991 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.178907 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.180873 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.181743 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.181849 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.181980 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:50.182084 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 14:46:50.182190 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":93,"threads_started":1}
I20260619 14:46:50.182521 19026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.175584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54102 (local address 127.0.98.2:43567)
0619 14:46:50.177035 (+ 1451us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:50.177049 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:50.177074 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:46:50.177152 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:50.177158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:50.177369 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:50.177676 (+ 307us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:50.177898 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.179269 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.180906 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.181478 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.181611 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.181787 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:50.182262 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 14:46:50.182383 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1346,"thread_start_us":177,"threads_started":1}
I20260619 14:46:50.182180 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.175767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45175 (local address 127.0.0.1:48328)
0619 14:46:50.176660 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:50.176675 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:50.176771 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:50.177702 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:50.177704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:50.177720 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:50.177851 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.177860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.179072 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.179077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.180729 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.180743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.181868 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.181871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.181950 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.181968 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:46:50.182034 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 14:46:50.182092 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":87,"thread_start_us":92,"threads_started":1}
I20260619 14:46:50.182883 19030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.176583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57452 (local address 127.0.98.3:45471)
0619 14:46:50.177095 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:50.177110 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:50.177131 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:46:50.177205 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:50.177209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:50.177408 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:50.177602 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:50.178102 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.179281 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.180932 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.181743 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.181850 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.181980 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:50.182596 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 14:46:50.182708 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":173,"threads_started":1}
I20260619 14:46:50.182498 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.176375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45471 (local address 127.0.0.1:57452)
0619 14:46:50.176707 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:50.176721 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:50.176786 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:50.177564 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:50.177567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:50.177604 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:50.177974 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.177980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.179499 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.179502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.180770 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.180780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.182050 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.182064 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.182166 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.182188 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:46:50.182260 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 14:46:50.182331 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":124,"spinlock_wait_cycles":4352,"thread_start_us":81,"threads_started":1}
I20260619 14:46:50.185544 18948 tablet_service.cc:1467] Processing CreateTablet for tablet 9d215beafe3f415e8b24a69cf8c8aa85 (DEFAULT_TABLE table=test-table [id=b17c21c6389049f2b67c1897a7b91525]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:50.185544 18814 tablet_service.cc:1467] Processing CreateTablet for tablet 9d215beafe3f415e8b24a69cf8c8aa85 (DEFAULT_TABLE table=test-table [id=b17c21c6389049f2b67c1897a7b91525]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:50.186970 18814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d215beafe3f415e8b24a69cf8c8aa85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:50.186993 18948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d215beafe3f415e8b24a69cf8c8aa85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:50.188788 18680 tablet_service.cc:1467] Processing CreateTablet for tablet 9d215beafe3f415e8b24a69cf8c8aa85 (DEFAULT_TABLE table=test-table [id=b17c21c6389049f2b67c1897a7b91525]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:50.189910 18680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d215beafe3f415e8b24a69cf8c8aa85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:50.193862 19031 tablet_bootstrap.cc:492] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428: Bootstrap starting.
I20260619 14:46:50.195173 19032 tablet_bootstrap.cc:492] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6: Bootstrap starting.
I20260619 14:46:50.196981 19033 tablet_bootstrap.cc:492] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f: Bootstrap starting.
I20260619 14:46:50.197055 19031 tablet_bootstrap.cc:654] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:50.197423 19032 tablet_bootstrap.cc:654] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:50.198468 19031 log.cc:824] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:50.198673 19032 log.cc:824] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:50.198963 19033 tablet_bootstrap.cc:654] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:50.199806 19033 log.cc:824] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:50.200832 19031 tablet_bootstrap.cc:492] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428: No bootstrap required, opened a new log
I20260619 14:46:50.200836 19032 tablet_bootstrap.cc:492] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6: No bootstrap required, opened a new log
I20260619 14:46:50.201028 19032 ts_tablet_manager.cc:1397] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:46:50.201045 19031 ts_tablet_manager.cc:1397] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260619 14:46:50.201639 19033 tablet_bootstrap.cc:492] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f: No bootstrap required, opened a new log
I20260619 14:46:50.201813 19033 ts_tablet_manager.cc:1397] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:46:50.205080 19031 raft_consensus.cc:348] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b222360b7fec4ad8919bcbb507d3a428" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "6ed99984067149ca8dc1c845245e516f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } } peers { permanent_uuid: "aa5cb4f3994445059b3b453cd3048ae6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45471 } }
I20260619 14:46:50.205080 19032 raft_consensus.cc:348] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b222360b7fec4ad8919bcbb507d3a428" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "6ed99984067149ca8dc1c845245e516f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } } peers { permanent_uuid: "aa5cb4f3994445059b3b453cd3048ae6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45471 } }
I20260619 14:46:50.205389 19031 raft_consensus.cc:374] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:50.205389 19032 raft_consensus.cc:374] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:50.205448 19032 raft_consensus.cc:729] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa5cb4f3994445059b3b453cd3048ae6, State: Initialized, Role: FOLLOWER
I20260619 14:46:50.205448 19031 raft_consensus.cc:729] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b222360b7fec4ad8919bcbb507d3a428, State: Initialized, Role: FOLLOWER
I20260619 14:46:50.205408 19033 raft_consensus.cc:348] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b222360b7fec4ad8919bcbb507d3a428" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "6ed99984067149ca8dc1c845245e516f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } } peers { permanent_uuid: "aa5cb4f3994445059b3b453cd3048ae6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45471 } }
I20260619 14:46:50.205687 19033 raft_consensus.cc:374] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:50.205747 19033 raft_consensus.cc:729] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed99984067149ca8dc1c845245e516f, State: Initialized, Role: FOLLOWER
I20260619 14:46:50.205972 19032 consensus_queue.cc:260] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b222360b7fec4ad8919bcbb507d3a428" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "6ed99984067149ca8dc1c845245e516f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } } peers { permanent_uuid: "aa5cb4f3994445059b3b453cd3048ae6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45471 } }
I20260619 14:46:50.205972 19031 consensus_queue.cc:260] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b222360b7fec4ad8919bcbb507d3a428" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "6ed99984067149ca8dc1c845245e516f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } } peers { permanent_uuid: "aa5cb4f3994445059b3b453cd3048ae6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45471 } }
I20260619 14:46:50.206234 19033 consensus_queue.cc:260] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b222360b7fec4ad8919bcbb507d3a428" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "6ed99984067149ca8dc1c845245e516f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } } peers { permanent_uuid: "aa5cb4f3994445059b3b453cd3048ae6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45471 } }
I20260619 14:46:50.206905 18880 heartbeater.cc:503] Master 127.0.98.62:42493 was elected leader, sending a full tablet report...
I20260619 14:46:50.206995 19014 heartbeater.cc:503] Master 127.0.98.62:42493 was elected leader, sending a full tablet report...
I20260619 14:46:50.207028 19031 ts_tablet_manager.cc:1428] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:46:50.207333 19033 ts_tablet_manager.cc:1428] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:46:50.207562 18746 heartbeater.cc:503] Master 127.0.98.62:42493 was elected leader, sending a full tablet report...
I20260619 14:46:50.208011 19032 ts_tablet_manager.cc:1428] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:46:50.217566 19039 raft_consensus.cc:482] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:50.217832 19039 raft_consensus.cc:504] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b222360b7fec4ad8919bcbb507d3a428" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "6ed99984067149ca8dc1c845245e516f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } } peers { permanent_uuid: "aa5cb4f3994445059b3b453cd3048ae6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45471 } }
I20260619 14:46:50.219076 19039 leader_election.cc:288] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b222360b7fec4ad8919bcbb507d3a428 (127.0.98.2:43567), aa5cb4f3994445059b3b453cd3048ae6 (127.0.98.3:45471)
I20260619 14:46:50.223214 19030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.219467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38027 (local address 127.0.98.3:45471)
0619 14:46:50.219594 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:50.219600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:50.219800 (+ 200us) server_negotiation.cc:407] Connection header received
0619 14:46:50.219964 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:50.219967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:50.220019 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:50.220103 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:50.220430 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.220971 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.222218 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.222629 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.222747 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.222823 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:50.223002 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 14:46:50.223075 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:46:50.223245 19026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.219422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54211 (local address 127.0.98.2:43567)
0619 14:46:50.219585 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:50.219592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:50.219913 (+ 321us) server_negotiation.cc:407] Connection header received
0619 14:46:50.220064 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:50.220068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:50.220113 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:50.220200 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:50.220491 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.220984 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.222243 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.222662 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.222788 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.222848 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:50.223034 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 14:46:50.223104 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 14:46:50.223272 19041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.219322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43567 (local address 127.0.98.1:54211)
0619 14:46:50.219808 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:50.219819 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:50.219949 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:50.220238 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:50.220241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:50.220257 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:50.220379 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.220384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.221100 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.221105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.222089 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.222099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.222789 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.222793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.222905 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.222928 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:46:50.223018 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:46:50.223090 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":52,"thread_start_us":137,"threads_started":1}
I20260619 14:46:50.223287 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.219300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45471 (local address 127.0.98.1:38027)
0619 14:46:50.219704 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:50.219716 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:50.219851 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:50.220114 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:50.220117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:50.220145 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:50.220317 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.220324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.221086 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.221091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.222090 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.222099 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.222756 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.222759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.222855 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.222896 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 14:46:50.222986 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:46:50.223062 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":63,"thread_start_us":165,"threads_started":1}
I20260619 14:46:50.224104 18968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d215beafe3f415e8b24a69cf8c8aa85" candidate_uuid: "6ed99984067149ca8dc1c845245e516f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa5cb4f3994445059b3b453cd3048ae6" is_pre_election: true
I20260619 14:46:50.224149 18834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d215beafe3f415e8b24a69cf8c8aa85" candidate_uuid: "6ed99984067149ca8dc1c845245e516f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b222360b7fec4ad8919bcbb507d3a428" is_pre_election: true
I20260619 14:46:50.224433 18968 raft_consensus.cc:2454] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed99984067149ca8dc1c845245e516f in term 0.
I20260619 14:46:50.224472 18834 raft_consensus.cc:2454] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed99984067149ca8dc1c845245e516f in term 0.
I20260619 14:46:50.224982 18634 leader_election.cc:302] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6ed99984067149ca8dc1c845245e516f, aa5cb4f3994445059b3b453cd3048ae6, b222360b7fec4ad8919bcbb507d3a428; no voters:
I20260619 14:46:50.225283 19039 raft_consensus.cc:2781] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:46:50.225385 19039 raft_consensus.cc:482] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:46:50.225431 19039 raft_consensus.cc:3037] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:50.226598 19039 raft_consensus.cc:504] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b222360b7fec4ad8919bcbb507d3a428" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "6ed99984067149ca8dc1c845245e516f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } } peers { permanent_uuid: "aa5cb4f3994445059b3b453cd3048ae6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45471 } }
I20260619 14:46:50.226982 19039 leader_election.cc:288] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f [CANDIDATE]: Term 1 election: Requested vote from peers b222360b7fec4ad8919bcbb507d3a428 (127.0.98.2:43567), aa5cb4f3994445059b3b453cd3048ae6 (127.0.98.3:45471)
I20260619 14:46:50.227368 18834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d215beafe3f415e8b24a69cf8c8aa85" candidate_uuid: "6ed99984067149ca8dc1c845245e516f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b222360b7fec4ad8919bcbb507d3a428"
I20260619 14:46:50.227562 18834 raft_consensus.cc:3037] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:50.227569 18968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d215beafe3f415e8b24a69cf8c8aa85" candidate_uuid: "6ed99984067149ca8dc1c845245e516f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa5cb4f3994445059b3b453cd3048ae6"
I20260619 14:46:50.227874 18968 raft_consensus.cc:3037] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:50.229269 18968 raft_consensus.cc:2454] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed99984067149ca8dc1c845245e516f in term 1.
I20260619 14:46:50.229315 18834 raft_consensus.cc:2454] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed99984067149ca8dc1c845245e516f in term 1.
I20260619 14:46:50.229612 18634 leader_election.cc:302] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed99984067149ca8dc1c845245e516f, aa5cb4f3994445059b3b453cd3048ae6; no voters:
I20260619 14:46:50.229815 19039 raft_consensus.cc:2781] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:50.230085 19039 raft_consensus.cc:686] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f [term 1 LEADER]: Becoming Leader. State: Replica: 6ed99984067149ca8dc1c845245e516f, State: Running, Role: LEADER
I20260619 14:46:50.230492 19039 consensus_queue.cc:237] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b222360b7fec4ad8919bcbb507d3a428" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } } peers { permanent_uuid: "6ed99984067149ca8dc1c845245e516f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } } peers { permanent_uuid: "aa5cb4f3994445059b3b453cd3048ae6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45471 } }
I20260619 14:46:50.234159 18553 catalog_manager.cc:5495] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed99984067149ca8dc1c845245e516f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6ed99984067149ca8dc1c845245e516f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b222360b7fec4ad8919bcbb507d3a428" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ed99984067149ca8dc1c845245e516f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa5cb4f3994445059b3b453cd3048ae6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45471 } health_report { overall_health: UNKNOWN } } }
I20260619 14:46:50.253016 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.248204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48330 (local address 127.0.98.1:45175)
0619 14:46:50.248350 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:50.248357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:50.248375 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:46:50.248462 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:50.248464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:50.248522 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:50.248642 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:50.248896 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.249513 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.251050 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.251305 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.251429 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.251874 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:50.252433 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:50.252616 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:50.252686 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 14:46:50.252847 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:46:50.260828 18968 raft_consensus.cc:1261] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6 [term 1 FOLLOWER]: Refusing update from remote peer 6ed99984067149ca8dc1c845245e516f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:50.261706 19042 consensus_queue.cc:1035] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa5cb4f3994445059b3b453cd3048ae6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:50.261996 18834 raft_consensus.cc:1261] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428 [term 1 FOLLOWER]: Refusing update from remote peer 6ed99984067149ca8dc1c845245e516f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:50.262975 19042 consensus_queue.cc:1035] T 9d215beafe3f415e8b24a69cf8c8aa85 P 6ed99984067149ca8dc1c845245e516f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b222360b7fec4ad8919bcbb507d3a428" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:50.271353 19046 mvcc.cc:205] Tried to move back new op lower bound from 7298582160420290560 to 7298582160307294208. Current Snapshot: MvccSnapshot[applied={T|T < 7298582160420290560}]
I20260619 14:46:50.274309 19047 mvcc.cc:205] Tried to move back new op lower bound from 7298582160420290560 to 7298582160307294208. Current Snapshot: MvccSnapshot[applied={T|T < 7298582160420290560 or (T in {7298582160420290560})}]
I20260619 14:46:50.284483 18553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:50.284667 18553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:50.287278 18553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2365703ec7424c6d9f84c47a762e701a: Sending DeleteTablet for 3 replicas of tablet 9d215beafe3f415e8b24a69cf8c8aa85
I20260619 14:46:50.288415 18948 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d215beafe3f415e8b24a69cf8c8aa85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57452
I20260619 14:46:50.288544 18814 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d215beafe3f415e8b24a69cf8c8aa85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54102
I20260619 14:46:50.289021 19055 tablet_replica.cc:330] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6: stopping tablet replica
I20260619 14:46:50.289124 19056 tablet_replica.cc:330] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428: stopping tablet replica
I20260619 14:46:50.289126 18680 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d215beafe3f415e8b24a69cf8c8aa85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48328
I20260619 14:46:50.289486 19056 raft_consensus.cc:2229] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:50.289513 19055 raft_consensus.cc:2229] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:50.289848 19056 raft_consensus.cc:2258] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:50.289916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 18617
I20260619 14:46:50.290068 19055 raft_consensus.cc:2258] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:50.292558 19055 ts_tablet_manager.cc:1905] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:50.292768 19056 ts_tablet_manager.cc:1905] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:50.295372 19056 ts_tablet_manager.cc:1918] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:50.295521 19056 log.cc:1196] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/9d215beafe3f415e8b24a69cf8c8aa85
I20260619 14:46:50.295858 19056 ts_tablet_manager.cc:1939] T 9d215beafe3f415e8b24a69cf8c8aa85 P b222360b7fec4ad8919bcbb507d3a428: Deleting consensus metadata
I20260619 14:46:50.296963 19055 ts_tablet_manager.cc:1918] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:50.297079 19055 log.cc:1196] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/9d215beafe3f415e8b24a69cf8c8aa85
I20260619 14:46:50.297449 19055 ts_tablet_manager.cc:1939] T 9d215beafe3f415e8b24a69cf8c8aa85 P aa5cb4f3994445059b3b453cd3048ae6: Deleting consensus metadata
I20260619 14:46:50.297648 18541 catalog_manager.cc:4841] TS b222360b7fec4ad8919bcbb507d3a428 (127.0.98.2:43567): tablet 9d215beafe3f415e8b24a69cf8c8aa85 (table test-table [id=b17c21c6389049f2b67c1897a7b91525]) successfully deleted
W20260619 14:46:50.298869 18540 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45175 (error 108)
I20260619 14:46:50.299013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 18751
I20260619 14:46:50.299391 18540 catalog_manager.cc:4841] TS aa5cb4f3994445059b3b453cd3048ae6 (127.0.98.3:45471): tablet 9d215beafe3f415e8b24a69cf8c8aa85 (table test-table [id=b17c21c6389049f2b67c1897a7b91525]) successfully deleted
I20260619 14:46:50.301302 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.300926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45175 (local address 127.0.0.1:48334)
0619 14:46:50.301108 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:50.301195 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 14:46:50.301486 18540 catalog_manager.cc:4584] TS 6ed99984067149ca8dc1c845245e516f (127.0.98.1:45175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d215beafe3f415e8b24a69cf8c8aa85: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45175: connect: Connection refused (error 111)
I20260619 14:46:50.304735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 18885
I20260619 14:46:50.310479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 18523
2026-06-19T14:46:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2368 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T14:46:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:50Z Disabled control of system clock
I20260619 14:46:50.342152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36391
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:46:50.454188 19062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:50.454495 19062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:50.454532 19062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:50.458698 19062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:50.458793 19062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:50.458811 19062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:50.458829 19062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:50.458844 19062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:50.463757 19062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36391
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:50.465284 19062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:50.466747 19062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:50.475971 19070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:50.476114 19062 server_base.cc:1034] running on GCE node
W20260619 14:46:50.476048 19067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:50.476013 19068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:50.476924 19062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:50.478282 19062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:50.479558 19062 hybrid_clock.cc:648] HybridClock initialized: now 1781880410479514 us; error 57 us; skew 500 ppm
I20260619 14:46:50.481808 19062 webserver.cc:460] Webserver started at http://127.0.98.62:46167/ using document root <none> and password file <none>
I20260619 14:46:50.482481 19062 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:50.482554 19062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:50.482806 19062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:50.485306 19062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "69a61c054def4cb5aea35c38c1e67301"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-cr22"
server_key: "b4c5e9c51714132b6b03780c1eb85573"
server_key_iv: "65be10cf55ad16e96616b42a8bbb6f11"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.485987 19062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "69a61c054def4cb5aea35c38c1e67301"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-cr22"
server_key: "b4c5e9c51714132b6b03780c1eb85573"
server_key_iv: "65be10cf55ad16e96616b42a8bbb6f11"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.490645 19062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 14:46:50.494177 19076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:50.496337 19062 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 14:46:50.496626 19062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "69a61c054def4cb5aea35c38c1e67301"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-cr22"
server_key: "b4c5e9c51714132b6b03780c1eb85573"
server_key_iv: "65be10cf55ad16e96616b42a8bbb6f11"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.496956 19062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:50.509716 19062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:50.510486 19062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:50.510708 19062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:50.518790 19062 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36391
I20260619 14:46:50.518829 19128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36391 every 8 connection(s)
I20260619 14:46:50.519855 19062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:50.523155 19129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:50.528920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 19062
I20260619 14:46:50.529072 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:50.529347 392 external_mini_cluster.cc:1428] Setting key 9eefc3ef3d3e39014129522634927f59
I20260619 14:46:50.529395 19129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301: Bootstrap starting.
I20260619 14:46:50.532157 19129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:50.532927 19129 log.cc:824] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:50.535018 19129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301: No bootstrap required, opened a new log
I20260619 14:46:50.538496 19129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69a61c054def4cb5aea35c38c1e67301" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36391 } }
I20260619 14:46:50.538820 19129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:50.538894 19129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a61c054def4cb5aea35c38c1e67301, State: Initialized, Role: FOLLOWER
I20260619 14:46:50.539351 19129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69a61c054def4cb5aea35c38c1e67301" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36391 } }
I20260619 14:46:50.539525 19129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:50.539616 19129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:50.539712 19129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:50.539954 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.530912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49674 (local address 127.0.98.62:36391)
0619 14:46:50.531715 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:50.531731 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:50.531771 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:46:50.532619 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:50.532639 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:50.533130 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:50.533526 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:50.533972 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.535116 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.536176 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.536709 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.536901 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.537848 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:50.537854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:50.537868 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:50.538290 (+ 422us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:50.538349 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:50.538370 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:50.538489 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:50.538617 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 14:46:50.538922 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":128,"threads_started":1}
I20260619 14:46:50.541152 19129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69a61c054def4cb5aea35c38c1e67301" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36391 } }
I20260619 14:46:50.541563 19129 leader_election.cc:302] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69a61c054def4cb5aea35c38c1e67301; no voters:
I20260619 14:46:50.541930 19129 leader_election.cc:288] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:50.542042 19134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:50.542286 19134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301 [term 1 LEADER]: Becoming Leader. State: Replica: 69a61c054def4cb5aea35c38c1e67301, State: Running, Role: LEADER
I20260619 14:46:50.542621 19134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69a61c054def4cb5aea35c38c1e67301" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36391 } }
I20260619 14:46:50.543300 19129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:50.544453 19136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69a61c054def4cb5aea35c38c1e67301. Latest consensus state: current_term: 1 leader_uuid: "69a61c054def4cb5aea35c38c1e67301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69a61c054def4cb5aea35c38c1e67301" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36391 } } }
I20260619 14:46:50.544612 19136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:50.545382 19142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:50.545775 19135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69a61c054def4cb5aea35c38c1e67301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69a61c054def4cb5aea35c38c1e67301" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36391 } } }
I20260619 14:46:50.545893 19135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:50.549240 19142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:50.555784 19142 catalog_manager.cc:1347] Generated new cluster ID: e46c4018ea7345a886d4e2626d7ec5e5
I20260619 14:46:50.555874 19142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:50.566538 19142 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:50.567555 19142 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:50.573589 19142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301: Generated new TSK 0
I20260619 14:46:50.574378 19142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:46:50.585685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36391
--builtin_ntp_servers=127.0.98.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:46:50.698901 19153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:50.699206 19153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:50.699280 19153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:50.703626 19153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:50.703711 19153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:50.703810 19153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:50.708585 19153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:50.709705 19153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:50.711138 19153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:50.721146 19161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:50.721163 19158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:50.721352 19153 server_base.cc:1034] running on GCE node
W20260619 14:46:50.721140 19159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:50.721932 19153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:50.722711 19153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:50.723901 19153 hybrid_clock.cc:648] HybridClock initialized: now 1781880410723841 us; error 74 us; skew 500 ppm
I20260619 14:46:50.725951 19153 webserver.cc:460] Webserver started at http://127.0.98.1:33769/ using document root <none> and password file <none>
I20260619 14:46:50.726552 19153 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:50.726650 19153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:50.726886 19153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:50.729029 19153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "ed2bcd53329745ce93e06b2f95a4d4ad"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-cr22"
server_key: "d7d039725429a1fb5d5398d13735184d"
server_key_iv: "c48507ef858c0fe5a0279eef9832b288"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.729610 19153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ed2bcd53329745ce93e06b2f95a4d4ad"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-cr22"
server_key: "d7d039725429a1fb5d5398d13735184d"
server_key_iv: "c48507ef858c0fe5a0279eef9832b288"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.733482 19153 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 14:46:50.736006 19167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:50.737355 19153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:46:50.737493 19153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "ed2bcd53329745ce93e06b2f95a4d4ad"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-cr22"
server_key: "d7d039725429a1fb5d5398d13735184d"
server_key_iv: "c48507ef858c0fe5a0279eef9832b288"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.737612 19153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:50.759011 19153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:50.759780 19153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:50.759990 19153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:50.760620 19153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:50.761612 19153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:50.761686 19153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:50.761757 19153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:50.761803 19153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:50.771030 19153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33365
I20260619 14:46:50.771052 19280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33365 every 8 connection(s)
I20260619 14:46:50.771981 19153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:50.773321 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 19153
I20260619 14:46:50.773453 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:46:50.773720 392 external_mini_cluster.cc:1428] Setting key fdfa13587e038bd17779b2fb1d1f3267
I20260619 14:46:50.775872 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36391
--builtin_ntp_servers=127.0.98.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:46:50.782502 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.774100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59803 (local address 127.0.98.62:36391)
0619 14:46:50.774245 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:50.774250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:50.775564 (+ 1314us) server_negotiation.cc:407] Connection header received
0619 14:46:50.777063 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:50.777066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:50.777134 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:50.777251 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:50.778345 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.778944 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.780290 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.780528 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.780648 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.781916 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:50.781927 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:50.781929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:50.781943 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:50.781963 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:50.781971 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:50.782053 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:50.782284 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 14:46:50.782353 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:46:50.783491 19283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.774649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36391 (local address 127.0.98.1:59803)
0619 14:46:50.775364 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:50.775417 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:50.776840 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:50.777425 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:50.777432 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:50.777578 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:50.778086 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.778098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.779092 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.779096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.780151 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.780162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.780698 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.780702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.780868 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.781593 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:46:50.781595 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:50.781662 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:46:50.781738 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:50.782056 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:50.782068 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:50.782075 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:46:50.782284 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 14:46:50.782587 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":143,"threads_started":1}
I20260619 14:46:50.784945 19281 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36391
I20260619 14:46:50.785255 19281 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:50.786093 19281 heartbeater.cc:511] Master 127.0.98.62:36391 requested a full tablet report, sending...
I20260619 14:46:50.787830 19093 ts_manager.cc:194] Registered new tserver with Master: ed2bcd53329745ce93e06b2f95a4d4ad (127.0.98.1:33365)
I20260619 14:46:50.789094 19093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:59803
W20260619 14:46:50.889705 19284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:50.890012 19284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:50.890079 19284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:50.894194 19284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:50.894289 19284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:50.894377 19284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:50.899757 19284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:50.900908 19284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:50.902446 19284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:50.912281 19292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:50.912345 19289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:50.912475 19284 server_base.cc:1034] running on GCE node
W20260619 14:46:50.912366 19290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:50.912904 19284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:50.913553 19284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:50.914778 19284 hybrid_clock.cc:648] HybridClock initialized: now 1781880410914764 us; error 29 us; skew 500 ppm
I20260619 14:46:50.916685 19284 webserver.cc:460] Webserver started at http://127.0.98.2:44811/ using document root <none> and password file <none>
I20260619 14:46:50.917286 19284 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:50.917379 19284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:50.917608 19284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:50.919728 19284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "78037f1e3da4436cb2d1d15d35469624"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-cr22"
server_key: "237b8b7d3f80f70572bbe3bd0f4bc24d"
server_key_iv: "81d9e8fe3dd816a5241d3b459b8ab1ad"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.920298 19284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "78037f1e3da4436cb2d1d15d35469624"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-cr22"
server_key: "237b8b7d3f80f70572bbe3bd0f4bc24d"
server_key_iv: "81d9e8fe3dd816a5241d3b459b8ab1ad"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.924050 19284 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:46:50.926666 19298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:50.928112 19284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:50.928258 19284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "78037f1e3da4436cb2d1d15d35469624"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-cr22"
server_key: "237b8b7d3f80f70572bbe3bd0f4bc24d"
server_key_iv: "81d9e8fe3dd816a5241d3b459b8ab1ad"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.928381 19284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:50.952796 19284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:50.953553 19284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:50.953773 19284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:50.954380 19284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:50.955334 19284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:50.955406 19284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:50.955490 19284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:50.955535 19284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:50.964679 19284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35053
I20260619 14:46:50.964737 19411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35053 every 8 connection(s)
I20260619 14:46:50.965619 19284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:50.973187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 19284
I20260619 14:46:50.973331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:46:50.973608 392 external_mini_cluster.cc:1428] Setting key 0951a15715aadd2f5891c9972561e867
I20260619 14:46:50.975616 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.967630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52281 (local address 127.0.98.62:36391)
0619 14:46:50.967864 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:50.967870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:50.969008 (+ 1138us) server_negotiation.cc:407] Connection header received
0619 14:46:50.970333 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:50.970336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:50.970386 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:50.970470 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:50.971347 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.972045 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.973351 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.973618 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.973745 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.975080 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:50.975084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:50.975087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:50.975101 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:50.975125 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:50.975140 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:50.975221 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:50.975383 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 14:46:50.975465 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260619 14:46:50.975806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36391
--builtin_ntp_servers=127.0.98.20:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:46:50.976547 19414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.968108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36391 (local address 127.0.98.2:52281)
0619 14:46:50.968843 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:50.968883 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:50.970114 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:50.970622 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:50.970629 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:50.970737 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:50.971207 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.971219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.972169 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.972173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.973191 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.973204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.973874 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.973877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.974011 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.974778 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:46:50.974779 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:50.974816 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:46:50.974915 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:50.975223 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:50.975234 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:50.975242 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:46:50.975382 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 14:46:50.975691 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":148,"threads_started":1}
I20260619 14:46:50.977689 19412 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36391
I20260619 14:46:50.977960 19412 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:50.978576 19412 heartbeater.cc:511] Master 127.0.98.62:36391 requested a full tablet report, sending...
I20260619 14:46:50.979812 19093 ts_manager.cc:194] Registered new tserver with Master: 78037f1e3da4436cb2d1d15d35469624 (127.0.98.2:35053)
I20260619 14:46:50.980448 19093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:52281
W20260619 14:46:51.096585 19415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:51.096892 19415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:51.096966 19415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:51.100997 19415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:51.101100 19415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:51.101204 19415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:51.106166 19415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:51.107336 19415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:51.108793 19415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:51.118710 19421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:51.118798 19420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:51.118769 19415 server_base.cc:1034] running on GCE node
W20260619 14:46:51.118719 19423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:51.119340 19415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:51.120071 19415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:51.121274 19415 hybrid_clock.cc:648] HybridClock initialized: now 1781880411121255 us; error 29 us; skew 500 ppm
I20260619 14:46:51.123349 19415 webserver.cc:460] Webserver started at http://127.0.98.3:35939/ using document root <none> and password file <none>
I20260619 14:46:51.123994 19415 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:51.124094 19415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:51.124337 19415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:51.126430 19415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "b111aaef465d446984021aeae891d30a"
format_stamp: "Formatted at 2026-06-19 14:46:51 on dist-test-slave-cr22"
server_key: "dec6d83a5611d9caeb4718515ca3d6cb"
server_key_iv: "7f5d11df1bf70bcbdce3f0fcccd0c3a5"
server_key_version: "encryptionkey@0"
I20260619 14:46:51.127007 19415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b111aaef465d446984021aeae891d30a"
format_stamp: "Formatted at 2026-06-19 14:46:51 on dist-test-slave-cr22"
server_key: "dec6d83a5611d9caeb4718515ca3d6cb"
server_key_iv: "7f5d11df1bf70bcbdce3f0fcccd0c3a5"
server_key_version: "encryptionkey@0"
I20260619 14:46:51.130779 19415 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:46:51.133496 19429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:51.134897 19415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:46:51.135078 19415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "b111aaef465d446984021aeae891d30a"
format_stamp: "Formatted at 2026-06-19 14:46:51 on dist-test-slave-cr22"
server_key: "dec6d83a5611d9caeb4718515ca3d6cb"
server_key_iv: "7f5d11df1bf70bcbdce3f0fcccd0c3a5"
server_key_version: "encryptionkey@0"
I20260619 14:46:51.135215 19415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:51.162637 19415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:51.163398 19415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:51.163630 19415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:51.164245 19415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:51.165195 19415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:51.165267 19415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:51.165349 19415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:51.165405 19415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:51.174882 19415 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32979
I20260619 14:46:51.175022 19542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32979 every 8 connection(s)
I20260619 14:46:51.175904 19415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:51.183641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 19415
I20260619 14:46:51.183779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:46:51.184082 392 external_mini_cluster.cc:1428] Setting key f4ecf2107c3bf3e0c16d327b7689fce1
I20260619 14:46:51.185940 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.177792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54149 (local address 127.0.98.62:36391)
0619 14:46:51.177931 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:51.177935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:51.178982 (+ 1047us) server_negotiation.cc:407] Connection header received
0619 14:46:51.180567 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:51.180570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:51.180633 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:51.180733 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:51.181632 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.182169 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.183359 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.183730 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.183853 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:51.185307 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:51.185311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:51.185314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:51.185330 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:51.185356 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:51.185370 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:51.185454 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:51.185734 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 14:46:51.185800 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:46:51.186956 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.178112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36391 (local address 127.0.98.3:54149)
0619 14:46:51.178806 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:51.178846 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:51.180339 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:51.180906 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:51.180913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:51.180999 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:51.181486 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:51.181498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.182293 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.182298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:51.183183 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:51.183192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.183939 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.183943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:51.184208 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:51.184932 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:46:51.184933 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:51.184973 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:46:51.185095 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:51.185486 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:51.185495 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:51.185502 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:46:51.185648 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 14:46:51.185911 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":108,"threads_started":1}
I20260619 14:46:51.188102 19543 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36391
I20260619 14:46:51.188359 19543 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:51.188933 19543 heartbeater.cc:511] Master 127.0.98.62:36391 requested a full tablet report, sending...
I20260619 14:46:51.189970 19093 ts_manager.cc:194] Registered new tserver with Master: b111aaef465d446984021aeae891d30a (127.0.98.3:32979)
I20260619 14:46:51.190544 19093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54149
I20260619 14:46:51.198385 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:46:51.205072 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.200395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49684 (local address 127.0.98.62:36391)
0619 14:46:51.200555 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:51.200560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:51.200864 (+ 304us) server_negotiation.cc:407] Connection header received
0619 14:46:51.200959 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:51.200962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:51.201009 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:51.201070 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:51.201404 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.201997 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.202983 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.203294 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.203408 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:51.204415 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:51.204418 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:51.204420 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:51.204431 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:51.204450 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:51.204459 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:51.204559 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:51.204868 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 14:46:51.204913 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:46:51.208058 19093 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:46:51.211318 19093 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:46:51.226534 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.221165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32979 (local address 127.0.0.1:55480)
0619 14:46:51.221985 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:51.221997 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:51.222078 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:51.222722 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:51.222725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:51.222742 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:51.222824 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:51.222831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.224131 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.224138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:51.225195 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:51.225212 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.225990 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.225993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:51.226096 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:51.226131 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 14:46:51.226248 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 14:46:51.226375 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":1,"thread_start_us":114,"threads_started":1}
I20260619 14:46:51.226538 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.221332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55480 (local address 127.0.98.3:32979)
0619 14:46:51.221696 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:51.221707 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:51.222170 (+ 463us) server_negotiation.cc:407] Connection header received
0619 14:46:51.222262 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:51.222269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:51.222429 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:51.222583 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:51.222963 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.224025 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.225326 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.225863 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.225992 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:51.226189 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:51.226272 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 14:46:51.226367 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":125,"threads_started":1}
I20260619 14:46:51.226864 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.220716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51414 (local address 127.0.98.1:33365)
0619 14:46:51.221138 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:51.221150 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:51.221593 (+ 443us) server_negotiation.cc:407] Connection header received
0619 14:46:51.221679 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:51.221684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:51.221902 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:51.222052 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:51.222537 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.223491 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.225116 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.225933 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.226031 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:51.226188 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:51.226609 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 14:46:51.226702 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":141,"threads_started":1}
I20260619 14:46:51.227020 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.220675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.0.98.2:35053)
0619 14:46:51.221351 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:51.221363 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:51.221726 (+ 363us) server_negotiation.cc:407] Connection header received
0619 14:46:51.221956 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:51.221964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:51.222127 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:51.222278 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:51.222642 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.223781 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.225192 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.225933 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.226020 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:51.226188 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:51.226764 (+ 576us) server_negotiation.cc:299] Negotiation successful
0619 14:46:51.226850 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":157,"threads_started":1}
I20260619 14:46:51.227253 19554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.220398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33365 (local address 127.0.0.1:51414)
0619 14:46:51.221215 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:51.221270 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:51.221421 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:51.222241 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:51.222247 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:51.222270 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:51.222410 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:51.222417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.223604 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.223609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:51.224974 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:51.224985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.226264 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.226268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:51.226409 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:51.226468 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 14:46:51.226533 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:46:51.227101 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":528,"thread_start_us":108,"threads_started":1}
I20260619 14:46:51.227528 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.220427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35053 (local address 127.0.0.1:41510)
0619 14:46:51.221623 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:51.221638 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:51.221801 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:51.222413 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:51.222415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:51.222430 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:51.222525 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:51.222533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.223896 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.223900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:51.225067 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:51.225075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.226276 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.226278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:51.226404 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:51.226462 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 14:46:51.226533 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 14:46:51.226622 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"mutex_wait_us":70,"thread_start_us":83,"threads_started":1}
I20260619 14:46:51.229945 19346 tablet_service.cc:1467] Processing CreateTablet for tablet 44725b4d0a3a485b9ba9e0ee03c7b593 (DEFAULT_TABLE table=test-table [id=a6176271b33d49fba96a743efbda264f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:51.230003 19215 tablet_service.cc:1467] Processing CreateTablet for tablet 44725b4d0a3a485b9ba9e0ee03c7b593 (DEFAULT_TABLE table=test-table [id=a6176271b33d49fba96a743efbda264f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:51.230000 19477 tablet_service.cc:1467] Processing CreateTablet for tablet 44725b4d0a3a485b9ba9e0ee03c7b593 (DEFAULT_TABLE table=test-table [id=a6176271b33d49fba96a743efbda264f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:51.230948 19346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44725b4d0a3a485b9ba9e0ee03c7b593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:51.230976 19215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44725b4d0a3a485b9ba9e0ee03c7b593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:51.230976 19477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44725b4d0a3a485b9ba9e0ee03c7b593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:51.237586 19560 tablet_bootstrap.cc:492] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624: Bootstrap starting.
I20260619 14:46:51.238525 19562 tablet_bootstrap.cc:492] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a: Bootstrap starting.
I20260619 14:46:51.239467 19561 tablet_bootstrap.cc:492] T 44725b4d0a3a485b9ba9e0ee03c7b593 P ed2bcd53329745ce93e06b2f95a4d4ad: Bootstrap starting.
I20260619 14:46:51.240162 19560 tablet_bootstrap.cc:654] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:51.240754 19562 tablet_bootstrap.cc:654] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:51.241026 19560 log.cc:824] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:51.241523 19561 tablet_bootstrap.cc:654] T 44725b4d0a3a485b9ba9e0ee03c7b593 P ed2bcd53329745ce93e06b2f95a4d4ad: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:51.241544 19562 log.cc:824] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:51.242419 19561 log.cc:824] T 44725b4d0a3a485b9ba9e0ee03c7b593 P ed2bcd53329745ce93e06b2f95a4d4ad: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:51.243597 19562 tablet_bootstrap.cc:492] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a: No bootstrap required, opened a new log
I20260619 14:46:51.243839 19562 ts_tablet_manager.cc:1397] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:46:51.244130 19560 tablet_bootstrap.cc:492] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624: No bootstrap required, opened a new log
I20260619 14:46:51.244318 19560 ts_tablet_manager.cc:1397] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20260619 14:46:51.244390 19561 tablet_bootstrap.cc:492] T 44725b4d0a3a485b9ba9e0ee03c7b593 P ed2bcd53329745ce93e06b2f95a4d4ad: No bootstrap required, opened a new log
I20260619 14:46:51.244557 19561 ts_tablet_manager.cc:1397] T 44725b4d0a3a485b9ba9e0ee03c7b593 P ed2bcd53329745ce93e06b2f95a4d4ad: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:46:51.247298 19562 raft_consensus.cc:348] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2bcd53329745ce93e06b2f95a4d4ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33365 } } peers { permanent_uuid: "78037f1e3da4436cb2d1d15d35469624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "b111aaef465d446984021aeae891d30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32979 } }
I20260619 14:46:51.247596 19562 raft_consensus.cc:374] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:51.247659 19562 raft_consensus.cc:729] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b111aaef465d446984021aeae891d30a, State: Initialized, Role: FOLLOWER
I20260619 14:46:51.248100 19562 consensus_queue.cc:260] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2bcd53329745ce93e06b2f95a4d4ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33365 } } peers { permanent_uuid: "78037f1e3da4436cb2d1d15d35469624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "b111aaef465d446984021aeae891d30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32979 } }
I20260619 14:46:51.248353 19560 raft_consensus.cc:348] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2bcd53329745ce93e06b2f95a4d4ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33365 } } peers { permanent_uuid: "78037f1e3da4436cb2d1d15d35469624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "b111aaef465d446984021aeae891d30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32979 } }
I20260619 14:46:51.248594 19560 raft_consensus.cc:374] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:51.248651 19560 raft_consensus.cc:729] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78037f1e3da4436cb2d1d15d35469624, State: Initialized, Role: FOLLOWER
I20260619 14:46:51.248652 19561 raft_consensus.cc:348] T 44725b4d0a3a485b9ba9e0ee03c7b593 P ed2bcd53329745ce93e06b2f95a4d4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2bcd53329745ce93e06b2f95a4d4ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33365 } } peers { permanent_uuid: "78037f1e3da4436cb2d1d15d35469624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "b111aaef465d446984021aeae891d30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32979 } }
I20260619 14:46:51.248876 19561 raft_consensus.cc:374] T 44725b4d0a3a485b9ba9e0ee03c7b593 P ed2bcd53329745ce93e06b2f95a4d4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:51.248874 19543 heartbeater.cc:503] Master 127.0.98.62:36391 was elected leader, sending a full tablet report...
I20260619 14:46:51.248926 19561 raft_consensus.cc:729] T 44725b4d0a3a485b9ba9e0ee03c7b593 P ed2bcd53329745ce93e06b2f95a4d4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2bcd53329745ce93e06b2f95a4d4ad, State: Initialized, Role: FOLLOWER
I20260619 14:46:51.249110 19560 consensus_queue.cc:260] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2bcd53329745ce93e06b2f95a4d4ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33365 } } peers { permanent_uuid: "78037f1e3da4436cb2d1d15d35469624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "b111aaef465d446984021aeae891d30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32979 } }
I20260619 14:46:51.249279 19561 consensus_queue.cc:260] T 44725b4d0a3a485b9ba9e0ee03c7b593 P ed2bcd53329745ce93e06b2f95a4d4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2bcd53329745ce93e06b2f95a4d4ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33365 } } peers { permanent_uuid: "78037f1e3da4436cb2d1d15d35469624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "b111aaef465d446984021aeae891d30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32979 } }
I20260619 14:46:51.250030 19560 ts_tablet_manager.cc:1428] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260619 14:46:51.250061 19561 ts_tablet_manager.cc:1428] T 44725b4d0a3a485b9ba9e0ee03c7b593 P ed2bcd53329745ce93e06b2f95a4d4ad: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:46:51.250100 19562 ts_tablet_manager.cc:1428] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:46:51.250357 19412 heartbeater.cc:503] Master 127.0.98.62:36391 was elected leader, sending a full tablet report...
I20260619 14:46:51.250430 19281 heartbeater.cc:503] Master 127.0.98.62:36391 was elected leader, sending a full tablet report...
W20260619 14:46:51.273294 19282 tablet.cc:2369] T 44725b4d0a3a485b9ba9e0ee03c7b593 P ed2bcd53329745ce93e06b2f95a4d4ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:46:51.314522 19567 raft_consensus.cc:482] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:51.314774 19567 raft_consensus.cc:504] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2bcd53329745ce93e06b2f95a4d4ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33365 } } peers { permanent_uuid: "78037f1e3da4436cb2d1d15d35469624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "b111aaef465d446984021aeae891d30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32979 } }
I20260619 14:46:51.316058 19567 leader_election.cc:288] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed2bcd53329745ce93e06b2f95a4d4ad (127.0.98.1:33365), b111aaef465d446984021aeae891d30a (127.0.98.3:32979)
I20260619 14:46:51.320346 19414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.316353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32979 (local address 127.0.98.2:45645)
0619 14:46:51.316510 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:51.316528 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:51.316637 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:51.316991 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:51.316994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:51.317010 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:51.317131 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:51.317138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.318195 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.318198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:51.319097 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:51.319106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.319916 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.319920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:51.320023 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:51.320062 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 14:46:51.320121 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 14:46:51.320206 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 14:46:51.320405 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.316353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33365 (local address 127.0.98.2:36039)
0619 14:46:51.316940 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:51.316954 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:51.317077 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:51.317374 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:51.317376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:51.317395 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:51.317509 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:51.317516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.318455 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.318457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:51.319261 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:51.319271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.319947 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.319951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:51.320069 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:51.320086 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:46:51.320192 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 14:46:51.320254 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":200,"thread_start_us":154,"threads_started":1}
I20260619 14:46:51.320410 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.316471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45645 (local address 127.0.98.3:32979)
0619 14:46:51.316589 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:51.316595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:51.316616 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:46:51.316812 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:51.316818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:51.316875 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:51.316981 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:51.317264 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.318094 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.319234 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.319783 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.319903 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:51.319971 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:51.320161 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 14:46:51.320243 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 14:46:51.320417 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.316473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36039 (local address 127.0.98.1:33365)
0619 14:46:51.316597 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:51.316602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:51.317048 (+ 446us) server_negotiation.cc:407] Connection header received
0619 14:46:51.317180 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:51.317183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:51.317262 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:51.317341 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:51.317636 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.318344 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.319388 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.319794 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.319922 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:51.319984 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:51.320231 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 14:46:51.320302 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:46:51.321038 19497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44725b4d0a3a485b9ba9e0ee03c7b593" candidate_uuid: "78037f1e3da4436cb2d1d15d35469624" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b111aaef465d446984021aeae891d30a" is_pre_election: true
I20260619 14:46:51.321205 19235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44725b4d0a3a485b9ba9e0ee03c7b593" candidate_uuid: "78037f1e3da4436cb2d1d15d35469624" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2bcd53329745ce93e06b2f95a4d4ad" is_pre_election: true
I20260619 14:46:51.321359 19497 raft_consensus.cc:2454] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78037f1e3da4436cb2d1d15d35469624 in term 0.
I20260619 14:46:51.321506 19235 raft_consensus.cc:2454] T 44725b4d0a3a485b9ba9e0ee03c7b593 P ed2bcd53329745ce93e06b2f95a4d4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78037f1e3da4436cb2d1d15d35469624 in term 0.
I20260619 14:46:51.321849 19300 leader_election.cc:302] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78037f1e3da4436cb2d1d15d35469624, b111aaef465d446984021aeae891d30a; no voters:
I20260619 14:46:51.322135 19567 raft_consensus.cc:2781] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:46:51.322209 19567 raft_consensus.cc:482] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:46:51.322240 19567 raft_consensus.cc:3037] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:51.323293 19567 raft_consensus.cc:504] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2bcd53329745ce93e06b2f95a4d4ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33365 } } peers { permanent_uuid: "78037f1e3da4436cb2d1d15d35469624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "b111aaef465d446984021aeae891d30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32979 } }
I20260619 14:46:51.323709 19567 leader_election.cc:288] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [CANDIDATE]: Term 1 election: Requested vote from peers ed2bcd53329745ce93e06b2f95a4d4ad (127.0.98.1:33365), b111aaef465d446984021aeae891d30a (127.0.98.3:32979)
I20260619 14:46:51.324039 19235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44725b4d0a3a485b9ba9e0ee03c7b593" candidate_uuid: "78037f1e3da4436cb2d1d15d35469624" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2bcd53329745ce93e06b2f95a4d4ad"
I20260619 14:46:51.324162 19235 raft_consensus.cc:3037] T 44725b4d0a3a485b9ba9e0ee03c7b593 P ed2bcd53329745ce93e06b2f95a4d4ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:51.324223 19497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44725b4d0a3a485b9ba9e0ee03c7b593" candidate_uuid: "78037f1e3da4436cb2d1d15d35469624" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b111aaef465d446984021aeae891d30a"
I20260619 14:46:51.324371 19497 raft_consensus.cc:3037] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:51.325258 19235 raft_consensus.cc:2454] T 44725b4d0a3a485b9ba9e0ee03c7b593 P ed2bcd53329745ce93e06b2f95a4d4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78037f1e3da4436cb2d1d15d35469624 in term 1.
I20260619 14:46:51.325456 19497 raft_consensus.cc:2454] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78037f1e3da4436cb2d1d15d35469624 in term 1.
I20260619 14:46:51.325647 19299 leader_election.cc:302] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78037f1e3da4436cb2d1d15d35469624, ed2bcd53329745ce93e06b2f95a4d4ad; no voters:
I20260619 14:46:51.325831 19567 raft_consensus.cc:2781] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:51.326066 19567 raft_consensus.cc:686] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [term 1 LEADER]: Becoming Leader. State: Replica: 78037f1e3da4436cb2d1d15d35469624, State: Running, Role: LEADER
I20260619 14:46:51.326458 19567 consensus_queue.cc:237] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2bcd53329745ce93e06b2f95a4d4ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33365 } } peers { permanent_uuid: "78037f1e3da4436cb2d1d15d35469624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } } peers { permanent_uuid: "b111aaef465d446984021aeae891d30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32979 } }
I20260619 14:46:51.329690 19093 catalog_manager.cc:5495] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78037f1e3da4436cb2d1d15d35469624 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "78037f1e3da4436cb2d1d15d35469624" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2bcd53329745ce93e06b2f95a4d4ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78037f1e3da4436cb2d1d15d35469624" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b111aaef465d446984021aeae891d30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32979 } health_report { overall_health: UNKNOWN } } }
I20260619 14:46:51.351781 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.347994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41514 (local address 127.0.98.2:35053)
0619 14:46:51.348159 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:51.348163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:51.348179 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:46:51.348259 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:51.348267 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:51.348325 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:51.348432 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:51.348695 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.349302 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.350238 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.350465 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.350554 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:51.350847 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:51.351258 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:51.351369 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:51.351612 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 14:46:51.351649 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 14:46:51.358301 19235 raft_consensus.cc:1261] T 44725b4d0a3a485b9ba9e0ee03c7b593 P ed2bcd53329745ce93e06b2f95a4d4ad [term 1 FOLLOWER]: Refusing update from remote peer 78037f1e3da4436cb2d1d15d35469624: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:51.358342 19497 raft_consensus.cc:1261] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a [term 1 FOLLOWER]: Refusing update from remote peer 78037f1e3da4436cb2d1d15d35469624: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:51.359094 19570 consensus_queue.cc:1035] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b111aaef465d446984021aeae891d30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:51.359323 19567 consensus_queue.cc:1035] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2bcd53329745ce93e06b2f95a4d4ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:51.366647 19573 mvcc.cc:205] Tried to move back new op lower bound from 7298582164915081216 to 7298582164795736064. Current Snapshot: MvccSnapshot[applied={T|T < 7298582164915081216}]
I20260619 14:46:51.367225 19575 mvcc.cc:205] Tried to move back new op lower bound from 7298582164915081216 to 7298582164795736064. Current Snapshot: MvccSnapshot[applied={T|T < 7298582164915081216}]
I20260619 14:46:51.369504 19574 mvcc.cc:205] Tried to move back new op lower bound from 7298582164915081216 to 7298582164795736064. Current Snapshot: MvccSnapshot[applied={T|T < 7298582164915081216}]
I20260619 14:46:51.375849 19093 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:51.376042 19093 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:51.378556 19093 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69a61c054def4cb5aea35c38c1e67301: Sending DeleteTablet for 3 replicas of tablet 44725b4d0a3a485b9ba9e0ee03c7b593
I20260619 14:46:51.379383 19215 tablet_service.cc:1514] Processing DeleteTablet for tablet 44725b4d0a3a485b9ba9e0ee03c7b593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:51 UTC) from {username='slave'} at 127.0.0.1:51414
I20260619 14:46:51.379820 19477 tablet_service.cc:1514] Processing DeleteTablet for tablet 44725b4d0a3a485b9ba9e0ee03c7b593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:51 UTC) from {username='slave'} at 127.0.0.1:55480
I20260619 14:46:51.379966 19583 tablet_replica.cc:330] T 44725b4d0a3a485b9ba9e0ee03c7b593 P ed2bcd53329745ce93e06b2f95a4d4ad: stopping tablet replica
I20260619 14:46:51.379942 19346 tablet_service.cc:1514] Processing DeleteTablet for tablet 44725b4d0a3a485b9ba9e0ee03c7b593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:51 UTC) from {username='slave'} at 127.0.0.1:41510
I20260619 14:46:51.380295 19583 raft_consensus.cc:2229] T 44725b4d0a3a485b9ba9e0ee03c7b593 P ed2bcd53329745ce93e06b2f95a4d4ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:51.380563 19585 tablet_replica.cc:330] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624: stopping tablet replica
I20260619 14:46:51.380617 19583 raft_consensus.cc:2258] T 44725b4d0a3a485b9ba9e0ee03c7b593 P ed2bcd53329745ce93e06b2f95a4d4ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:51.380862 19584 tablet_replica.cc:330] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a: stopping tablet replica
I20260619 14:46:51.380978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 19153
I20260619 14:46:51.381178 19585 raft_consensus.cc:2229] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:46:51.381541 19585 raft_consensus.cc:2258] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:51.381979 19584 raft_consensus.cc:2229] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:51.382257 19584 raft_consensus.cc:2258] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:51.383941 19584 ts_tablet_manager.cc:1905] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:51.386181 19584 ts_tablet_manager.cc:1918] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:51.386296 19584 log.cc:1196] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/44725b4d0a3a485b9ba9e0ee03c7b593
I20260619 14:46:51.386623 19584 ts_tablet_manager.cc:1939] T 44725b4d0a3a485b9ba9e0ee03c7b593 P b111aaef465d446984021aeae891d30a: Deleting consensus metadata
I20260619 14:46:51.387717 19585 ts_tablet_manager.cc:1905] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:51.387889 19078 catalog_manager.cc:4841] TS b111aaef465d446984021aeae891d30a (127.0.98.3:32979): tablet 44725b4d0a3a485b9ba9e0ee03c7b593 (table test-table [id=a6176271b33d49fba96a743efbda264f]) successfully deleted
W20260619 14:46:51.389710 19077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33365 (error 108)
I20260619 14:46:51.390244 19585 ts_tablet_manager.cc:1918] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:51.390424 19585 log.cc:1196] T 44725b4d0a3a485b9ba9e0ee03c7b593 P 78037f1e3da4436cb2d1d15d35469624: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/44725b4d0a3a485b9ba9e0ee03c7b593
I20260619 14:46:51.390476 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 19284
I20260619 14:46:51.390821 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.390448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33365 (local address 127.0.0.1:51426)
0619 14:46:51.390641 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:51.390725 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260619 14:46:51.391003 19077 catalog_manager.cc:4584] TS ed2bcd53329745ce93e06b2f95a4d4ad (127.0.98.1:33365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44725b4d0a3a485b9ba9e0ee03c7b593: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33365: connect: Connection refused (error 111)
I20260619 14:46:51.395613 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 19415
I20260619 14:46:51.396478 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.396146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35053 (local address 127.0.0.1:41530)
0619 14:46:51.396299 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:51.396369 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 14:46:51.401410 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 19062
2026-06-19T14:46:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1089 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:46:51 dist-test-slave-cr22 krb5kdc[19589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:46:51 dist-test-slave-cr22 krb5kdc[19589](info): set up 2 sockets
Jun 19 14:46:51 dist-test-slave-cr22 krb5kdc[19589](info): commencing operation
krb5kdc: starting...
W20260619 14:46:52.475979 392 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:46:52 dist-test-slave-cr22 krb5kdc[19589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:46:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:46:52.613245 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39073
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:52.724021 19602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:52.724325 19602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:52.724404 19602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:52.729032 19602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:52.729100 19602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:52.729120 19602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:52.729136 19602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:52.729198 19602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:52.734587 19602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:52.735759 19602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:52.736871 19602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:46:52.745832 19602 server_base.cc:1034] running on GCE node
W20260619 14:46:52.745791 19608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:52.745791 19607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:52.746272 19610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:52.746943 19602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:52.748185 19602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:52.749393 19602 hybrid_clock.cc:648] HybridClock initialized: now 1781880412749366 us; error 37 us; skew 500 ppm
Jun 19 14:46:52 dist-test-slave-cr22 krb5kdc[19589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:52.755697 19602 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:46:52.756806 19602 webserver.cc:460] Webserver started at http://127.0.98.62:39861/ using document root <none> and password file <none>
I20260619 14:46:52.757551 19602 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:52.757658 19602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:52.757912 19602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:52.760475 19602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "c86481a623e04b239034b19aa53f71c9"
format_stamp: "Formatted at 2026-06-19 14:46:52 on dist-test-slave-cr22"
server_key: "abf79666ee82ac9138e010858436022f"
server_key_iv: "4ddf0df1c0a8dd75c5bf98dc5c4078b7"
server_key_version: "encryptionkey@0"
I20260619 14:46:52.761081 19602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "c86481a623e04b239034b19aa53f71c9"
format_stamp: "Formatted at 2026-06-19 14:46:52 on dist-test-slave-cr22"
server_key: "abf79666ee82ac9138e010858436022f"
server_key_iv: "4ddf0df1c0a8dd75c5bf98dc5c4078b7"
server_key_version: "encryptionkey@0"
I20260619 14:46:52.765554 19602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:46:52.768281 19617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:52.769649 19602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:52.769800 19602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "c86481a623e04b239034b19aa53f71c9"
format_stamp: "Formatted at 2026-06-19 14:46:52 on dist-test-slave-cr22"
server_key: "abf79666ee82ac9138e010858436022f"
server_key_iv: "4ddf0df1c0a8dd75c5bf98dc5c4078b7"
server_key_version: "encryptionkey@0"
I20260619 14:46:52.769924 19602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:52.790383 19602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:52.793915 19602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:52.794153 19602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:52.801261 19602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39073
I20260619 14:46:52.801321 19669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39073 every 8 connection(s)
I20260619 14:46:52.802286 19602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:52.805137 19670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:52.810559 19670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9: Bootstrap starting.
I20260619 14:46:52.810702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 19602
I20260619 14:46:52.810827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:52.811120 392 external_mini_cluster.cc:1428] Setting key 81ddbc4cc4a886bb12ca3aafae1c2805
I20260619 14:46:52.813304 19670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:52.814085 19670 log.cc:824] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:52.815997 19670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9: No bootstrap required, opened a new log
I20260619 14:46:52.819344 19670 raft_consensus.cc:348] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86481a623e04b239034b19aa53f71c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39073 } }
I20260619 14:46:52.819610 19670 raft_consensus.cc:374] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:52.819700 19670 raft_consensus.cc:729] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86481a623e04b239034b19aa53f71c9, State: Initialized, Role: FOLLOWER
Jun 19 14:46:52 dist-test-slave-cr22 krb5kdc[19589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:52.820204 19670 consensus_queue.cc:260] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86481a623e04b239034b19aa53f71c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39073 } }
I20260619 14:46:52.820334 19670 raft_consensus.cc:388] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:52.820415 19670 raft_consensus.cc:482] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:52.820505 19670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:52.821626 19670 raft_consensus.cc:504] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86481a623e04b239034b19aa53f71c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39073 } }
I20260619 14:46:52.821983 19670 leader_election.cc:302] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c86481a623e04b239034b19aa53f71c9; no voters:
I20260619 14:46:52.822319 19670 leader_election.cc:288] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:52.822389 19675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:52.822582 19675 raft_consensus.cc:686] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9 [term 1 LEADER]: Becoming Leader. State: Replica: c86481a623e04b239034b19aa53f71c9, State: Running, Role: LEADER
I20260619 14:46:52.822940 19675 consensus_queue.cc:237] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86481a623e04b239034b19aa53f71c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39073 } }
I20260619 14:46:52.823606 19670 sys_catalog.cc:564] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:52.824542 19676 sys_catalog.cc:455] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c86481a623e04b239034b19aa53f71c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86481a623e04b239034b19aa53f71c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39073 } } }
I20260619 14:46:52.824678 19676 sys_catalog.cc:458] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:52.824980 19677 sys_catalog.cc:455] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c86481a623e04b239034b19aa53f71c9. Latest consensus state: current_term: 1 leader_uuid: "c86481a623e04b239034b19aa53f71c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86481a623e04b239034b19aa53f71c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39073 } } }
I20260619 14:46:52.825062 19677 sys_catalog.cc:458] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:52.825950 19684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:52.826659 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:52.812603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34792 (local address 127.0.98.62:39073)
0619 14:46:52.813017 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:52.813033 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:52.813067 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:46:52.813852 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:52.813872 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:52.814249 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:52.814594 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:52.815401 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:52.816494 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:52.817339 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:52.817832 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:52.818014 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:52.820731 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:52.820737 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:52.820750 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:52.820777 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:52.822810 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:52.823312 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:52.823318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:52.823324 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:52.823503 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:52.823800 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:52.823803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:52.823805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:52.824326 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:52.824505 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:52.825367 (+ 862us) server_negotiation.cc:299] Negotiation successful
0619 14:46:52.825709 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":145,"threads_started":1}
I20260619 14:46:52.829217 19684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:52.835745 19684 catalog_manager.cc:1347] Generated new cluster ID: 6d1b4d15d836454cb17f940455c203df
I20260619 14:46:52.835836 19684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:52.850102 19684 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:52.851176 19684 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:52.858101 19684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9: Generated new TSK 0
I20260619 14:46:52.858855 19684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:46:52.890973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39073
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:53.022040 19696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:53.022348 19696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:53.022435 19696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:53.027515 19696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:53.027614 19696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:53.027722 19696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:53.033495 19696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:53.034706 19696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:53.035863 19696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:53.046875 19702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:53.046896 19701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:53.046898 19704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:53.047317 19696 server_base.cc:1034] running on GCE node
I20260619 14:46:53.048002 19696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:53.048640 19696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:53.049863 19696 hybrid_clock.cc:648] HybridClock initialized: now 1781880413049837 us; error 42 us; skew 500 ppm
Jun 19 14:46:53 dist-test-slave-cr22 krb5kdc[19589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:53.053102 19696 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:46:53.054173 19696 webserver.cc:460] Webserver started at http://127.0.98.1:40605/ using document root <none> and password file <none>
I20260619 14:46:53.054775 19696 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:53.054841 19696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:53.055058 19696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:53.057271 19696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "e9fbcff1c0864dbaaf12037c251bd7a6"
format_stamp: "Formatted at 2026-06-19 14:46:53 on dist-test-slave-cr22"
server_key: "e147468dace379bd96f72d2010d88cda"
server_key_iv: "ab26667dbc7d30f242a344311cc9c8af"
server_key_version: "encryptionkey@0"
I20260619 14:46:53.058175 19696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e9fbcff1c0864dbaaf12037c251bd7a6"
format_stamp: "Formatted at 2026-06-19 14:46:53 on dist-test-slave-cr22"
server_key: "e147468dace379bd96f72d2010d88cda"
server_key_iv: "ab26667dbc7d30f242a344311cc9c8af"
server_key_version: "encryptionkey@0"
I20260619 14:46:53.063123 19696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 14:46:53.066679 19711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:53.068279 19696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:53.068392 19696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "e9fbcff1c0864dbaaf12037c251bd7a6"
format_stamp: "Formatted at 2026-06-19 14:46:53 on dist-test-slave-cr22"
server_key: "e147468dace379bd96f72d2010d88cda"
server_key_iv: "ab26667dbc7d30f242a344311cc9c8af"
server_key_version: "encryptionkey@0"
I20260619 14:46:53.068554 19696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:53.086333 19696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:53.089851 19696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:53.090080 19696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:53.090723 19696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:53.091661 19696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:53.091712 19696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:53.091753 19696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:53.091812 19696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:53.103724 19696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35767
I20260619 14:46:53.103955 19824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35767 every 8 connection(s)
I20260619 14:46:53.104758 19696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:53.109334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 19696
I20260619 14:46:53.109480 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:46:53.109804 392 external_mini_cluster.cc:1428] Setting key cb6d6ca786c95397bcdd070a3af2a6f0
Jun 19 14:46:53 dist-test-slave-cr22 krb5kdc[19589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:46:53.123096 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.107429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43695 (local address 127.0.98.62:39073)
0619 14:46:53.107656 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:53.107660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:53.108807 (+ 1147us) server_negotiation.cc:407] Connection header received
0619 14:46:53.110176 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:53.110180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:53.110263 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:53.110372 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:53.112568 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.113122 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.114476 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.114917 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.115093 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.118271 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:53.118275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:53.118278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:53.118311 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:53.120315 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:53.120930 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:53.120940 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:53.120942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:53.121023 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:53.121611 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:53.121619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:53.121623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:53.122040 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:53.122147 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:53.122828 (+ 681us) server_negotiation.cc:299] Negotiation successful
0619 14:46:53.122965 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:46:53.124310 19827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.107831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39073 (local address 127.0.98.1:43695)
0619 14:46:53.108626 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:53.108676 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:53.109921 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:53.110673 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:53.110681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:53.111137 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:53.112378 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.112403 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.113280 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.113286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.114344 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.114354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.115144 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.115148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.115309 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.116192 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:53.116229 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:53.118064 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:53.120443 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:53.120448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:53.120461 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:53.120807 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:53.121139 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:53.121141 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:53.121143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:53.121301 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:53.122159 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:53.122164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:53.122496 (+ 332us) client_negotiation.cc:769] Sending connection context
0619 14:46:53.122731 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 14:46:53.123053 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":114,"threads_started":1}
I20260619 14:46:53.125741 19825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39073
I20260619 14:46:53.126031 19825 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:53.126709 19825 heartbeater.cc:511] Master 127.0.98.62:39073 requested a full tablet report, sending...
I20260619 14:46:53.128512 19634 ts_manager.cc:194] Registered new tserver with Master: e9fbcff1c0864dbaaf12037c251bd7a6 (127.0.98.1:35767)
I20260619 14:46:53.129729 19634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:46:53.143991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39073
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:53.255545 19830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:53.255892 19830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:53.255970 19830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:53.260102 19830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:53.260221 19830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:53.260318 19830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:53.265787 19830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:53.266988 19830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:53.268060 19830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:53.278587 19836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:53.278738 19838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:53.278774 19835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:53.279081 19830 server_base.cc:1034] running on GCE node
I20260619 14:46:53.279580 19830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:53.280277 19830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:53.281452 19830 hybrid_clock.cc:648] HybridClock initialized: now 1781880413281436 us; error 32 us; skew 500 ppm
Jun 19 14:46:53 dist-test-slave-cr22 krb5kdc[19589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:53.285030 19830 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:46:53.286020 19830 webserver.cc:460] Webserver started at http://127.0.98.2:39409/ using document root <none> and password file <none>
I20260619 14:46:53.286615 19830 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:53.286677 19830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:53.286896 19830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:53.289088 19830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "e5f6171e84684e1493af5239e2a7ba7d"
format_stamp: "Formatted at 2026-06-19 14:46:53 on dist-test-slave-cr22"
server_key: "ca91f74d0f52293d0cd4b7bf430b001d"
server_key_iv: "cd55c65e63b0c8f375a637a432857798"
server_key_version: "encryptionkey@0"
I20260619 14:46:53.289678 19830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e5f6171e84684e1493af5239e2a7ba7d"
format_stamp: "Formatted at 2026-06-19 14:46:53 on dist-test-slave-cr22"
server_key: "ca91f74d0f52293d0cd4b7bf430b001d"
server_key_iv: "cd55c65e63b0c8f375a637a432857798"
server_key_version: "encryptionkey@0"
I20260619 14:46:53.293625 19830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 14:46:53.296311 19845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:53.297655 19830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:53.297797 19830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "e5f6171e84684e1493af5239e2a7ba7d"
format_stamp: "Formatted at 2026-06-19 14:46:53 on dist-test-slave-cr22"
server_key: "ca91f74d0f52293d0cd4b7bf430b001d"
server_key_iv: "cd55c65e63b0c8f375a637a432857798"
server_key_version: "encryptionkey@0"
I20260619 14:46:53.297919 19830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:53.320345 19830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:53.323956 19830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:53.324173 19830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:53.324957 19830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:53.326128 19830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:53.326200 19830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:53.326269 19830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:53.326288 19830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:53.335397 19958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36297 every 8 connection(s)
I20260619 14:46:53.335549 19830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36297
I20260619 14:46:53.336668 19830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:53.341197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 19830
I20260619 14:46:53.341374 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:46:53.341670 392 external_mini_cluster.cc:1428] Setting key e0bbdd672578031726fe9d9569212a37
Jun 19 14:46:53 dist-test-slave-cr22 krb5kdc[19589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:46:53.352128 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.338747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36491 (local address 127.0.98.62:39073)
0619 14:46:53.338907 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:53.338912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:53.339991 (+ 1079us) server_negotiation.cc:407] Connection header received
0619 14:46:53.340949 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:53.340952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:53.341012 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:53.341112 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:53.342658 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.343346 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.344675 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.345170 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.345303 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.348137 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:53.348147 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:53.348150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:53.348179 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:53.349777 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:53.350401 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:53.350404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:53.350406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:53.350486 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:53.350888 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:53.350891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:53.350893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:53.351093 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:53.351219 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:53.351813 (+ 594us) server_negotiation.cc:299] Negotiation successful
0619 14:46:53.351945 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 14:46:53.353072 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.339084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39073 (local address 127.0.98.2:36491)
0619 14:46:53.339838 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:53.339880 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:53.340710 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:53.341346 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:53.341377 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:53.341852 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:53.342511 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.342528 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.343556 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.343561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.344535 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.344545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.345353 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.345356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.345505 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.346194 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:53.346212 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:53.347881 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:53.349920 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:53.349926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:53.349943 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:53.350274 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:53.350584 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:53.350587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:53.350590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:53.350740 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:53.351201 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:53.351208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:53.351562 (+ 354us) client_negotiation.cc:769] Sending connection context
0619 14:46:53.351758 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 14:46:53.352138 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":114,"threads_started":1}
I20260619 14:46:53.354504 19959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39073
I20260619 14:46:53.354794 19959 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:46:53.355577 19959 heartbeater.cc:511] Master 127.0.98.62:39073 requested a full tablet report, sending...
I20260619 14:46:53.356976 19634 ts_manager.cc:194] Registered new tserver with Master: e5f6171e84684e1493af5239e2a7ba7d (127.0.98.2:36297)
I20260619 14:46:53.357846 19634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:46:53.375864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39073
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:53.494807 19964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:53.495139 19964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:53.495214 19964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:53.500913 19964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:53.501027 19964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:53.501129 19964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:53.507265 19964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:53.508473 19964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:53.509574 19964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:53.520344 19970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:53.520364 19969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:53.520352 19972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:53.520809 19964 server_base.cc:1034] running on GCE node
I20260619 14:46:53.521314 19964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:53.521994 19964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:53.523340 19964 hybrid_clock.cc:648] HybridClock initialized: now 1781880413523322 us; error 38 us; skew 500 ppm
Jun 19 14:46:53 dist-test-slave-cr22 krb5kdc[19589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:53.526806 19964 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:46:53.527851 19964 webserver.cc:460] Webserver started at http://127.0.98.3:34379/ using document root <none> and password file <none>
I20260619 14:46:53.528445 19964 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:53.528539 19964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:53.528779 19964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:53.530898 19964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "97d9436e388a42d2b2cfe9229e0e2b35"
format_stamp: "Formatted at 2026-06-19 14:46:53 on dist-test-slave-cr22"
server_key: "059edd0f6458af8a7a7e0e3325a0a689"
server_key_iv: "b4223e55fd2a819760d19e0e9f1a710f"
server_key_version: "encryptionkey@0"
I20260619 14:46:53.531527 19964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "97d9436e388a42d2b2cfe9229e0e2b35"
format_stamp: "Formatted at 2026-06-19 14:46:53 on dist-test-slave-cr22"
server_key: "059edd0f6458af8a7a7e0e3325a0a689"
server_key_iv: "b4223e55fd2a819760d19e0e9f1a710f"
server_key_version: "encryptionkey@0"
I20260619 14:46:53.535388 19964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:46:53.538053 19979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:53.539410 19964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:46:53.539603 19964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "97d9436e388a42d2b2cfe9229e0e2b35"
format_stamp: "Formatted at 2026-06-19 14:46:53 on dist-test-slave-cr22"
server_key: "059edd0f6458af8a7a7e0e3325a0a689"
server_key_iv: "b4223e55fd2a819760d19e0e9f1a710f"
server_key_version: "encryptionkey@0"
I20260619 14:46:53.539729 19964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:53.576198 19964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:53.579303 19964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:53.579599 19964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:53.580271 19964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:53.581116 19964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:53.581192 19964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:53.581259 19964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:53.581311 19964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:53.591130 19964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32997
I20260619 14:46:53.591291 20092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32997 every 8 connection(s)
I20260619 14:46:53.592125 19964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:53.593690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 19964
I20260619 14:46:53.593828 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:46:53.594321 392 external_mini_cluster.cc:1428] Setting key 2fb4f7254e7285a0505424190f8a8ca3
Jun 19 14:46:53 dist-test-slave-cr22 krb5kdc[19589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:53.606848 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.594092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44631 (local address 127.0.98.62:39073)
0619 14:46:53.594257 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:53.594261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:53.595403 (+ 1142us) server_negotiation.cc:407] Connection header received
0619 14:46:53.596607 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:53.596611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:53.596666 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:53.596767 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:53.597912 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.598443 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.599665 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.599895 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.600006 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.602655 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:53.602665 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:53.602668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:53.602691 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:53.604771 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:53.605291 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:53.605296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:53.605298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:53.605379 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:53.605718 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:53.605723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:53.605725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:53.605963 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:53.606066 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:53.606613 (+ 547us) server_negotiation.cc:299] Negotiation successful
0619 14:46:53.606723 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:46:53.607962 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.594580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39073 (local address 127.0.98.3:44631)
0619 14:46:53.595174 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:53.595232 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:53.596371 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:53.596947 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:53.596954 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:53.597312 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:53.597775 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.597788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.598566 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.598570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.599538 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.599548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.600043 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.600047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.600225 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.600823 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:53.600839 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:53.602486 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:53.604904 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:53.604910 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:53.604924 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:53.605182 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:53.605479 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:53.605481 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:53.605483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:53.605622 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:53.606106 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:53.606111 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:53.606355 (+ 244us) client_negotiation.cc:769] Sending connection context
0619 14:46:53.606524 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 14:46:53.606784 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":150,"threads_started":1}
I20260619 14:46:53.609094 20093 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39073
I20260619 14:46:53.609360 20093 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:53.609802 20093 heartbeater.cc:511] Master 127.0.98.62:39073 requested a full tablet report, sending...
I20260619 14:46:53.610762 19633 ts_manager.cc:194] Registered new tserver with Master: 97d9436e388a42d2b2cfe9229e0e2b35 (127.0.98.3:32997)
I20260619 14:46:53.611320 19633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44631
I20260619 14:46:53.621169 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:46:53.630963 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.623577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.0.98.62:39073)
0619 14:46:53.623707 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:53.623714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:53.623732 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:46:53.623772 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:53.623775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:53.623817 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:53.623910 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:53.624694 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.625219 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.626102 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.626341 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.626455 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.627902 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:53.627906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:53.627909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:53.627933 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:53.629245 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:53.629694 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:53.629699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:53.629701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:53.629786 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:53.630066 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:53.630071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:53.630073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:53.630307 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:53.630449 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:53.630680 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 14:46:53.630799 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:46:53.634315 19633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:46:53.637414 19633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:46:53.651625 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.646123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35767 (local address 127.0.0.1:59078)
0619 14:46:53.646923 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:53.646936 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:53.647016 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:53.647750 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:53.647758 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:53.647782 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:53.647927 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.647934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.649067 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.649073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.650232 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.650241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.651027 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.651030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.651142 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.651183 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 14:46:53.651316 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 14:46:53.651441 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":191,"threads_started":1}
I20260619 14:46:53.651628 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.646851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59078 (local address 127.0.98.1:35767)
0619 14:46:53.647229 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:53.647237 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:53.647261 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:46:53.647343 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:53.647348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:53.647583 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:53.647732 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:53.648056 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.648955 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.650351 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.650883 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.651010 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.651160 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:53.651344 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 14:46:53.651470 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":73,"threads_started":1}
I20260619 14:46:53.654253 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.646165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48202 (local address 127.0.98.3:32997)
0619 14:46:53.647654 (+ 1489us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:53.647666 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:53.647699 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:46:53.647769 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:53.647773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:53.647932 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:53.648082 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:53.648463 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.649602 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.651270 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.652111 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.652238 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.652372 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:53.653992 (+ 1620us) server_negotiation.cc:299] Negotiation successful
0619 14:46:53.654089 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1390,"thread_start_us":153,"threads_started":1}
I20260619 14:46:53.654248 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.646797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36297 (local address 127.0.0.1:34786)
0619 14:46:53.647282 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:53.647296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:53.647375 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:53.648107 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:53.648109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:53.648125 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:53.648231 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.648238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.649698 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.649703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.651116 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.651129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.653802 (+ 2673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.653810 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.653900 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.653924 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:46:53.654005 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:46:53.654077 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":105,"thread_start_us":83,"threads_started":1}
I20260619 14:46:53.654488 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.646081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32997 (local address 127.0.0.1:48202)
0619 14:46:53.646480 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:53.646525 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:53.646648 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:53.648238 (+ 1590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:53.648241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:53.648258 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:53.648339 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.648345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.649745 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.649748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.651125 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.651131 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.653768 (+ 2637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.653774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.653844 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.653863 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:46:53.653959 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 14:46:53.654035 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":107,"thread_start_us":92,"threads_started":1}
I20260619 14:46:53.654801 19759 tablet_service.cc:1467] Processing CreateTablet for tablet f61e2964da1641898abebff1aa0d5840 (DEFAULT_TABLE table=test-table [id=a8d9b889847c4d178d8f29600779bff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:53.654618 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.647236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34786 (local address 127.0.98.2:36297)
0619 14:46:53.647684 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:53.647697 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:53.647730 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:46:53.647800 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:53.647805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:53.647993 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:53.648136 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:53.648345 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.649537 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.651262 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.652490 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.652598 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.652731 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:53.654174 (+ 1443us) server_negotiation.cc:299] Negotiation successful
0619 14:46:53.654273 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":117,"threads_started":1}
I20260619 14:46:53.655912 19759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61e2964da1641898abebff1aa0d5840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:53.657236 20027 tablet_service.cc:1467] Processing CreateTablet for tablet f61e2964da1641898abebff1aa0d5840 (DEFAULT_TABLE table=test-table [id=a8d9b889847c4d178d8f29600779bff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:53.658099 20027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61e2964da1641898abebff1aa0d5840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:53.657943 19893 tablet_service.cc:1467] Processing CreateTablet for tablet f61e2964da1641898abebff1aa0d5840 (DEFAULT_TABLE table=test-table [id=a8d9b889847c4d178d8f29600779bff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:53.658887 19893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61e2964da1641898abebff1aa0d5840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:53.663616 20110 tablet_bootstrap.cc:492] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6: Bootstrap starting.
I20260619 14:46:53.665052 20111 tablet_bootstrap.cc:492] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35: Bootstrap starting.
I20260619 14:46:53.665757 20112 tablet_bootstrap.cc:492] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d: Bootstrap starting.
I20260619 14:46:53.666260 20110 tablet_bootstrap.cc:654] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:53.667081 20111 tablet_bootstrap.cc:654] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:53.667785 20110 log.cc:824] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:53.668329 20112 tablet_bootstrap.cc:654] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:53.668349 20111 log.cc:824] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:53.669205 20112 log.cc:824] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:53.670024 20110 tablet_bootstrap.cc:492] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6: No bootstrap required, opened a new log
I20260619 14:46:53.670228 20110 ts_tablet_manager.cc:1397] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 14:46:53.670410 20111 tablet_bootstrap.cc:492] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35: No bootstrap required, opened a new log
I20260619 14:46:53.670588 20111 ts_tablet_manager.cc:1397] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 14:46:53.670903 20112 tablet_bootstrap.cc:492] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d: No bootstrap required, opened a new log
I20260619 14:46:53.671090 20112 ts_tablet_manager.cc:1397] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:46:53.674795 20111 raft_consensus.cc:348] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d9436e388a42d2b2cfe9229e0e2b35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } } peers { permanent_uuid: "e9fbcff1c0864dbaaf12037c251bd7a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35767 } } peers { permanent_uuid: "e5f6171e84684e1493af5239e2a7ba7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36297 } }
I20260619 14:46:53.674599 20112 raft_consensus.cc:348] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d9436e388a42d2b2cfe9229e0e2b35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } } peers { permanent_uuid: "e9fbcff1c0864dbaaf12037c251bd7a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35767 } } peers { permanent_uuid: "e5f6171e84684e1493af5239e2a7ba7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36297 } }
I20260619 14:46:53.675107 20111 raft_consensus.cc:374] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:53.675118 20112 raft_consensus.cc:374] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:53.675168 20111 raft_consensus.cc:729] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97d9436e388a42d2b2cfe9229e0e2b35, State: Initialized, Role: FOLLOWER
I20260619 14:46:53.675204 20112 raft_consensus.cc:729] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f6171e84684e1493af5239e2a7ba7d, State: Initialized, Role: FOLLOWER
I20260619 14:46:53.675271 20110 raft_consensus.cc:348] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d9436e388a42d2b2cfe9229e0e2b35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } } peers { permanent_uuid: "e9fbcff1c0864dbaaf12037c251bd7a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35767 } } peers { permanent_uuid: "e5f6171e84684e1493af5239e2a7ba7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36297 } }
I20260619 14:46:53.675559 20110 raft_consensus.cc:374] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:53.675662 20110 raft_consensus.cc:729] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fbcff1c0864dbaaf12037c251bd7a6, State: Initialized, Role: FOLLOWER
I20260619 14:46:53.675740 20111 consensus_queue.cc:260] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d9436e388a42d2b2cfe9229e0e2b35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } } peers { permanent_uuid: "e9fbcff1c0864dbaaf12037c251bd7a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35767 } } peers { permanent_uuid: "e5f6171e84684e1493af5239e2a7ba7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36297 } }
I20260619 14:46:53.675796 20112 consensus_queue.cc:260] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d9436e388a42d2b2cfe9229e0e2b35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } } peers { permanent_uuid: "e9fbcff1c0864dbaaf12037c251bd7a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35767 } } peers { permanent_uuid: "e5f6171e84684e1493af5239e2a7ba7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36297 } }
I20260619 14:46:53.676091 20110 consensus_queue.cc:260] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d9436e388a42d2b2cfe9229e0e2b35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } } peers { permanent_uuid: "e9fbcff1c0864dbaaf12037c251bd7a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35767 } } peers { permanent_uuid: "e5f6171e84684e1493af5239e2a7ba7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36297 } }
I20260619 14:46:53.676630 19959 heartbeater.cc:503] Master 127.0.98.62:39073 was elected leader, sending a full tablet report...
I20260619 14:46:53.676892 20112 ts_tablet_manager.cc:1428] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:46:53.677016 20110 ts_tablet_manager.cc:1428] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260619 14:46:53.677346 19825 heartbeater.cc:503] Master 127.0.98.62:39073 was elected leader, sending a full tablet report...
I20260619 14:46:53.677898 20111 ts_tablet_manager.cc:1428] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260619 14:46:53.678185 20093 heartbeater.cc:503] Master 127.0.98.62:39073 was elected leader, sending a full tablet report...
W20260619 14:46:53.837946 19960 tablet.cc:2369] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:46:53.843166 20094 tablet.cc:2369] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:46:53.856403 19826 tablet.cc:2369] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:46:53.872753 20118 raft_consensus.cc:482] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:53.872990 20118 raft_consensus.cc:504] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d9436e388a42d2b2cfe9229e0e2b35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } } peers { permanent_uuid: "e9fbcff1c0864dbaaf12037c251bd7a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35767 } } peers { permanent_uuid: "e5f6171e84684e1493af5239e2a7ba7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36297 } }
I20260619 14:46:53.874424 20118 leader_election.cc:288] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97d9436e388a42d2b2cfe9229e0e2b35 (127.0.98.3:32997), e5f6171e84684e1493af5239e2a7ba7d (127.0.98.2:36297)
I20260619 14:46:53.878947 20117 raft_consensus.cc:482] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:53.879228 20117 raft_consensus.cc:504] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d9436e388a42d2b2cfe9229e0e2b35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } } peers { permanent_uuid: "e9fbcff1c0864dbaaf12037c251bd7a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35767 } } peers { permanent_uuid: "e5f6171e84684e1493af5239e2a7ba7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36297 } }
I20260619 14:46:53.880486 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.874867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48189 (local address 127.0.98.3:32997)
0619 14:46:53.875008 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:53.875015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:53.875531 (+ 516us) server_negotiation.cc:407] Connection header received
0619 14:46:53.875596 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:53.875599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:53.875659 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:53.875759 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:53.876224 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.877125 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.879020 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.879694 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.879775 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.879842 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:53.880281 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 14:46:53.880358 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:46:53.880484 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.874837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35611 (local address 127.0.98.2:36297)
0619 14:46:53.875065 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:53.875071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:53.875308 (+ 237us) server_negotiation.cc:407] Connection header received
0619 14:46:53.875597 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:53.875600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:53.875700 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:53.875784 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:53.876327 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.877177 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.879020 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.879695 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.879775 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.879842 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:53.880281 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 14:46:53.880358 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260619 14:46:53.880906 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.874750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36297 (local address 127.0.98.1:35611)
0619 14:46:53.875188 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:53.875205 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:53.875348 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:53.875793 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:53.875795 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:53.875837 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:53.876210 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.876218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.877307 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.877311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.878867 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.878880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.879940 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.879944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.880103 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.880157 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 14:46:53.880607 (+ 450us) client_negotiation.cc:240] Negotiation successful
0619 14:46:53.880723 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":348,"spinlock_wait_cycles":9472,"thread_start_us":112,"threads_started":1}
I20260619 14:46:53.880906 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.874751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32997 (local address 127.0.98.1:48189)
0619 14:46:53.875286 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:53.875301 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:53.875381 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:53.875792 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:53.875795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:53.875836 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:53.876095 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.876103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.877307 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.877311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.878879 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.878885 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.879940 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.879944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.880119 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.880163 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 14:46:53.880607 (+ 444us) client_negotiation.cc:240] Negotiation successful
0619 14:46:53.880723 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":300,"spinlock_wait_cycles":31616,"thread_start_us":122,"threads_started":1}
I20260619 14:46:53.881484 19913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61e2964da1641898abebff1aa0d5840" candidate_uuid: "e9fbcff1c0864dbaaf12037c251bd7a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5f6171e84684e1493af5239e2a7ba7d" is_pre_election: true
I20260619 14:46:53.881719 19913 raft_consensus.cc:2454] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fbcff1c0864dbaaf12037c251bd7a6 in term 0.
I20260619 14:46:53.882053 20047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61e2964da1641898abebff1aa0d5840" candidate_uuid: "e9fbcff1c0864dbaaf12037c251bd7a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d9436e388a42d2b2cfe9229e0e2b35" is_pre_election: true
I20260619 14:46:53.882249 19712 leader_election.cc:302] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e5f6171e84684e1493af5239e2a7ba7d, e9fbcff1c0864dbaaf12037c251bd7a6; no voters:
I20260619 14:46:53.882364 20047 raft_consensus.cc:2454] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fbcff1c0864dbaaf12037c251bd7a6 in term 0.
I20260619 14:46:53.882580 20118 raft_consensus.cc:2781] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:46:53.882630 20117 leader_election.cc:288] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97d9436e388a42d2b2cfe9229e0e2b35 (127.0.98.3:32997), e9fbcff1c0864dbaaf12037c251bd7a6 (127.0.98.1:35767)
I20260619 14:46:53.882670 20118 raft_consensus.cc:482] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:46:53.882717 20118 raft_consensus.cc:3037] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:53.884001 20118 raft_consensus.cc:504] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d9436e388a42d2b2cfe9229e0e2b35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } } peers { permanent_uuid: "e9fbcff1c0864dbaaf12037c251bd7a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35767 } } peers { permanent_uuid: "e5f6171e84684e1493af5239e2a7ba7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36297 } }
I20260619 14:46:53.884404 20118 leader_election.cc:288] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 [CANDIDATE]: Term 1 election: Requested vote from peers 97d9436e388a42d2b2cfe9229e0e2b35 (127.0.98.3:32997), e5f6171e84684e1493af5239e2a7ba7d (127.0.98.2:36297)
I20260619 14:46:53.884716 20047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61e2964da1641898abebff1aa0d5840" candidate_uuid: "e9fbcff1c0864dbaaf12037c251bd7a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d9436e388a42d2b2cfe9229e0e2b35"
I20260619 14:46:53.884771 19913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61e2964da1641898abebff1aa0d5840" candidate_uuid: "e9fbcff1c0864dbaaf12037c251bd7a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5f6171e84684e1493af5239e2a7ba7d"
I20260619 14:46:53.884903 19913 raft_consensus.cc:3037] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:53.884917 20047 raft_consensus.cc:3037] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:53.886193 20122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.880694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32997 (local address 127.0.98.2:57841)
0619 14:46:53.881709 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:53.881724 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:53.881915 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:53.882293 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:53.882296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:53.882313 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:53.882418 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.882424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.883370 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.883373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.884665 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.884673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.885723 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.885727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.885829 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.885858 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:46:53.885948 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:46:53.886023 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"thread_start_us":101,"threads_started":1}
I20260619 14:46:53.886209 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.881067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57841 (local address 127.0.98.3:32997)
0619 14:46:53.881181 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:53.881185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:53.881813 (+ 628us) server_negotiation.cc:407] Connection header received
0619 14:46:53.882042 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:53.882047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:53.882095 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:53.882143 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:53.882563 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.883270 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.884850 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.885592 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.885703 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.885762 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:53.885971 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 14:46:53.886054 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:46:53.886211 19913 raft_consensus.cc:2454] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fbcff1c0864dbaaf12037c251bd7a6 in term 1.
I20260619 14:46:53.886283 20047 raft_consensus.cc:2454] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fbcff1c0864dbaaf12037c251bd7a6 in term 1.
I20260619 14:46:53.886613 19714 leader_election.cc:302] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97d9436e388a42d2b2cfe9229e0e2b35, e9fbcff1c0864dbaaf12037c251bd7a6; no voters:
I20260619 14:46:53.886636 20047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61e2964da1641898abebff1aa0d5840" candidate_uuid: "e5f6171e84684e1493af5239e2a7ba7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d9436e388a42d2b2cfe9229e0e2b35" is_pre_election: true
I20260619 14:46:53.886809 20047 raft_consensus.cc:2379] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5f6171e84684e1493af5239e2a7ba7d in current term 1: Already voted for candidate e9fbcff1c0864dbaaf12037c251bd7a6 in this term.
I20260619 14:46:53.886855 20118 raft_consensus.cc:2781] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:53.887132 20118 raft_consensus.cc:686] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 [term 1 LEADER]: Becoming Leader. State: Replica: e9fbcff1c0864dbaaf12037c251bd7a6, State: Running, Role: LEADER
I20260619 14:46:53.887502 20118 consensus_queue.cc:237] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d9436e388a42d2b2cfe9229e0e2b35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } } peers { permanent_uuid: "e9fbcff1c0864dbaaf12037c251bd7a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35767 } } peers { permanent_uuid: "e5f6171e84684e1493af5239e2a7ba7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36297 } }
I20260619 14:46:53.889497 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.881416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48855 (local address 127.0.98.1:35767)
0619 14:46:53.881516 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:53.881521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:53.882922 (+ 1401us) server_negotiation.cc:407] Connection header received
0619 14:46:53.883094 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:53.883098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:53.883140 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:53.883235 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:53.883521 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.884278 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.888314 (+ 4036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.888912 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.889038 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.889117 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:53.889317 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 14:46:53.889393 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 14:46:53.889503 20121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.881331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35767 (local address 127.0.98.2:48855)
0619 14:46:53.882846 (+ 1515us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:53.882860 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:53.882980 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:53.883245 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:53.883248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:53.883264 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:53.883373 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.883379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.884419 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.884421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.888214 (+ 3793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:53.888223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.889064 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.889067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:53.889145 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.889164 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:46:53.889282 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 14:46:53.889354 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1456,"thread_start_us":87,"threads_started":1}
I20260619 14:46:53.890133 19779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61e2964da1641898abebff1aa0d5840" candidate_uuid: "e5f6171e84684e1493af5239e2a7ba7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fbcff1c0864dbaaf12037c251bd7a6" is_pre_election: true
I20260619 14:46:53.890764 19849 leader_election.cc:302] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5f6171e84684e1493af5239e2a7ba7d; no voters: 97d9436e388a42d2b2cfe9229e0e2b35, e9fbcff1c0864dbaaf12037c251bd7a6
I20260619 14:46:53.891043 20117 raft_consensus.cc:2726] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 14:46:53.891142 19634 catalog_manager.cc:5495] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9fbcff1c0864dbaaf12037c251bd7a6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e9fbcff1c0864dbaaf12037c251bd7a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d9436e388a42d2b2cfe9229e0e2b35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9fbcff1c0864dbaaf12037c251bd7a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5f6171e84684e1493af5239e2a7ba7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36297 } health_report { overall_health: UNKNOWN } } }
I20260619 14:46:53.946964 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.943267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59086 (local address 127.0.98.1:35767)
0619 14:46:53.943396 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:53.943401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:53.943419 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:46:53.943605 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:53.943609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:53.943682 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:53.943795 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:53.943995 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.944612 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.945517 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:53.945735 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:53.945816 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:53.946086 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:53.946492 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:53.946579 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:53.946795 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 14:46:53.946848 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:46:53.953310 20047 raft_consensus.cc:1261] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35 [term 1 FOLLOWER]: Refusing update from remote peer e9fbcff1c0864dbaaf12037c251bd7a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:53.953349 19913 raft_consensus.cc:1261] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d [term 1 FOLLOWER]: Refusing update from remote peer e9fbcff1c0864dbaaf12037c251bd7a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:53.954025 20123 consensus_queue.cc:1035] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97d9436e388a42d2b2cfe9229e0e2b35" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:53.954231 20118 consensus_queue.cc:1035] T f61e2964da1641898abebff1aa0d5840 P e9fbcff1c0864dbaaf12037c251bd7a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5f6171e84684e1493af5239e2a7ba7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:53.962898 20128 mvcc.cc:205] Tried to move back new op lower bound from 7298582175544176640 to 7298582175286104064. Current Snapshot: MvccSnapshot[applied={T|T < 7298582175544176640}]
I20260619 14:46:53.970849 19634 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:53.971047 19634 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:53.973701 19634 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c86481a623e04b239034b19aa53f71c9: Sending DeleteTablet for 3 replicas of tablet f61e2964da1641898abebff1aa0d5840
I20260619 14:46:53.974474 19759 tablet_service.cc:1514] Processing DeleteTablet for tablet f61e2964da1641898abebff1aa0d5840 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59078
I20260619 14:46:53.974493 20027 tablet_service.cc:1514] Processing DeleteTablet for tablet f61e2964da1641898abebff1aa0d5840 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48202
I20260619 14:46:53.974671 19893 tablet_service.cc:1514] Processing DeleteTablet for tablet f61e2964da1641898abebff1aa0d5840 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34786
I20260619 14:46:53.974980 20136 tablet_replica.cc:330] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35: stopping tablet replica
I20260619 14:46:53.975204 20138 tablet_replica.cc:330] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d: stopping tablet replica
I20260619 14:46:53.975310 20136 raft_consensus.cc:2229] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:53.975490 20138 raft_consensus.cc:2229] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:53.975636 20136 raft_consensus.cc:2258] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:53.975759 20138 raft_consensus.cc:2258] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:53.975965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 19696
I20260619 14:46:53.977216 20136 ts_tablet_manager.cc:1905] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:53.977293 20138 ts_tablet_manager.cc:1905] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:53.979949 20138 ts_tablet_manager.cc:1918] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:53.979949 20136 ts_tablet_manager.cc:1918] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:53.980053 20138 log.cc:1196] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-1/wal/wals/f61e2964da1641898abebff1aa0d5840
I20260619 14:46:53.980058 20136 log.cc:1196] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/f61e2964da1641898abebff1aa0d5840
I20260619 14:46:53.980417 20136 ts_tablet_manager.cc:1939] T f61e2964da1641898abebff1aa0d5840 P 97d9436e388a42d2b2cfe9229e0e2b35: Deleting consensus metadata
I20260619 14:46:53.980820 20138 ts_tablet_manager.cc:1939] T f61e2964da1641898abebff1aa0d5840 P e5f6171e84684e1493af5239e2a7ba7d: Deleting consensus metadata
I20260619 14:46:53.981902 19620 catalog_manager.cc:4841] TS 97d9436e388a42d2b2cfe9229e0e2b35 (127.0.98.3:32997): tablet f61e2964da1641898abebff1aa0d5840 (table test-table [id=a8d9b889847c4d178d8f29600779bff9]) successfully deleted
I20260619 14:46:53.981986 19618 catalog_manager.cc:4841] TS e5f6171e84684e1493af5239e2a7ba7d (127.0.98.2:36297): tablet f61e2964da1641898abebff1aa0d5840 (table test-table [id=a8d9b889847c4d178d8f29600779bff9]) successfully deleted
W20260619 14:46:53.984128 19621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35767 (error 108)
I20260619 14:46:53.984311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 19830
I20260619 14:46:53.988344 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:53.988028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35767 (local address 127.0.0.1:59092)
0619 14:46:53.988157 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:53.988269 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 14:46:53.988636 19621 catalog_manager.cc:4584] TS e9fbcff1c0864dbaaf12037c251bd7a6 (127.0.98.1:35767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f61e2964da1641898abebff1aa0d5840: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35767: connect: Connection refused (error 111)
I20260619 14:46:53.990275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 19964
I20260619 14:46:53.995929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 19602
2026-06-19T14:46:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2594 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T14:46:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:54Z Disabled control of system clock
I20260619 14:46:54.026633 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32987
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:46:54.146198 20143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:54.146471 20143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:54.146509 20143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:54.150447 20143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:54.150506 20143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:54.150521 20143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:54.150538 20143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:54.150553 20143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:54.155225 20143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32987
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:54.156345 20143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:54.157646 20143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:54.166127 20148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:54.166136 20149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:54.166217 20151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:54.166272 20143 server_base.cc:1034] running on GCE node
I20260619 14:46:54.166982 20143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:54.168095 20143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:54.169309 20143 hybrid_clock.cc:648] HybridClock initialized: now 1781880414169294 us; error 31 us; skew 500 ppm
I20260619 14:46:54.171336 20143 webserver.cc:460] Webserver started at http://127.0.98.62:35391/ using document root <none> and password file <none>
I20260619 14:46:54.171975 20143 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:54.172070 20143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:54.172312 20143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:54.174386 20143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "8d1ff9b451bf4381a71cdb60a7563ef6"
format_stamp: "Formatted at 2026-06-19 14:46:54 on dist-test-slave-cr22"
server_key: "02e57e47bf503a2536b4f6ba55c12407"
server_key_iv: "83597c39e131de65ce19e7f0eb04e554"
server_key_version: "encryptionkey@0"
I20260619 14:46:54.174968 20143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d1ff9b451bf4381a71cdb60a7563ef6"
format_stamp: "Formatted at 2026-06-19 14:46:54 on dist-test-slave-cr22"
server_key: "02e57e47bf503a2536b4f6ba55c12407"
server_key_iv: "83597c39e131de65ce19e7f0eb04e554"
server_key_version: "encryptionkey@0"
I20260619 14:46:54.178678 20143 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:46:54.181147 20157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:54.182448 20143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:46:54.182600 20143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "8d1ff9b451bf4381a71cdb60a7563ef6"
format_stamp: "Formatted at 2026-06-19 14:46:54 on dist-test-slave-cr22"
server_key: "02e57e47bf503a2536b4f6ba55c12407"
server_key_iv: "83597c39e131de65ce19e7f0eb04e554"
server_key_version: "encryptionkey@0"
I20260619 14:46:54.182729 20143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:54.204505 20143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:54.205240 20143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:54.205452 20143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:54.212831 20143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32987
I20260619 14:46:54.212898 20209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32987 every 8 connection(s)
I20260619 14:46:54.213872 20143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:54.217337 20210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:54.223227 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 20143
I20260619 14:46:54.223331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:54.223677 392 external_mini_cluster.cc:1428] Setting key 28cf546d957a100f1c9edc907feb0e2d
I20260619 14:46:54.223668 20210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6: Bootstrap starting.
I20260619 14:46:54.226306 20210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:54.227063 20210 log.cc:824] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:54.228974 20210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6: No bootstrap required, opened a new log
I20260619 14:46:54.232542 20210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1ff9b451bf4381a71cdb60a7563ef6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32987 } }
I20260619 14:46:54.232801 20210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:54.232888 20210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d1ff9b451bf4381a71cdb60a7563ef6, State: Initialized, Role: FOLLOWER
I20260619 14:46:54.233227 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:54.225059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43784 (local address 127.0.98.62:32987)
0619 14:46:54.225755 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:54.225770 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:54.225806 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:46:54.226614 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:54.226638 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:54.227051 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:54.227469 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:54.227895 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.228834 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.229768 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.230245 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.230427 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:54.231322 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:54.231327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:54.231339 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:54.231820 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:54.231874 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:54.231894 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:54.232005 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:54.232120 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 14:46:54.232384 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":139,"threads_started":1}
I20260619 14:46:54.233403 20210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1ff9b451bf4381a71cdb60a7563ef6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32987 } }
I20260619 14:46:54.233562 20210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:54.233628 20210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:54.233719 20210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:54.234750 20210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1ff9b451bf4381a71cdb60a7563ef6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32987 } }
I20260619 14:46:54.235096 20210 leader_election.cc:302] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d1ff9b451bf4381a71cdb60a7563ef6; no voters:
I20260619 14:46:54.235484 20210 leader_election.cc:288] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:54.235626 20215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:54.235891 20215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6 [term 1 LEADER]: Becoming Leader. State: Replica: 8d1ff9b451bf4381a71cdb60a7563ef6, State: Running, Role: LEADER
I20260619 14:46:54.236217 20215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1ff9b451bf4381a71cdb60a7563ef6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32987 } }
I20260619 14:46:54.236748 20210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:54.238413 20216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d1ff9b451bf4381a71cdb60a7563ef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1ff9b451bf4381a71cdb60a7563ef6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32987 } } }
I20260619 14:46:54.238550 20216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:54.238612 20217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d1ff9b451bf4381a71cdb60a7563ef6. Latest consensus state: current_term: 1 leader_uuid: "8d1ff9b451bf4381a71cdb60a7563ef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d1ff9b451bf4381a71cdb60a7563ef6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32987 } } }
I20260619 14:46:54.238703 20217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:54.239338 20225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:54.242534 20225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:54.249029 20225 catalog_manager.cc:1347] Generated new cluster ID: 169f6768e6224d8cb44bd5017c6e8f23
I20260619 14:46:54.249261 20225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:54.286835 20225 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:54.288189 20225 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:54.299149 20225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6: Generated new TSK 0
I20260619 14:46:54.300052 20225 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:46:54.311179 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32987
--builtin_ntp_servers=127.0.98.20:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:46:54.427482 20234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:54.427755 20234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:54.427794 20234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:54.431957 20234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:54.432015 20234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:54.432072 20234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:54.436846 20234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:54.438047 20234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:54.439484 20234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:54.449327 20240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:54.449501 20234 server_base.cc:1034] running on GCE node
W20260619 14:46:54.449448 20239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:54.449327 20242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:54.450032 20234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:54.450700 20234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:54.451938 20234 hybrid_clock.cc:648] HybridClock initialized: now 1781880414451886 us; error 65 us; skew 500 ppm
I20260619 14:46:54.454349 20234 webserver.cc:460] Webserver started at http://127.0.98.1:44595/ using document root <none> and password file <none>
I20260619 14:46:54.454984 20234 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:54.455096 20234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:54.455348 20234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:54.457816 20234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-0/data/instance:
uuid: "f81e494bc993463eae313ea8ab3e8471"
format_stamp: "Formatted at 2026-06-19 14:46:54 on dist-test-slave-cr22"
server_key: "69f4160109eab1bc183550d74053513b"
server_key_iv: "aecd5f79be03b4cc2311988ce2822f3b"
server_key_version: "encryptionkey@0"
I20260619 14:46:54.458403 20234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f81e494bc993463eae313ea8ab3e8471"
format_stamp: "Formatted at 2026-06-19 14:46:54 on dist-test-slave-cr22"
server_key: "69f4160109eab1bc183550d74053513b"
server_key_iv: "aecd5f79be03b4cc2311988ce2822f3b"
server_key_version: "encryptionkey@0"
I20260619 14:46:54.462349 20234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 14:46:54.464999 20248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:54.466252 20234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:46:54.466392 20234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-0/wal
uuid: "f81e494bc993463eae313ea8ab3e8471"
format_stamp: "Formatted at 2026-06-19 14:46:54 on dist-test-slave-cr22"
server_key: "69f4160109eab1bc183550d74053513b"
server_key_iv: "aecd5f79be03b4cc2311988ce2822f3b"
server_key_version: "encryptionkey@0"
I20260619 14:46:54.466513 20234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:54.487825 20234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:54.488602 20234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:54.488811 20234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:54.489423 20234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:54.490393 20234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:54.490468 20234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:54.490546 20234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:54.490599 20234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:54.502151 20234 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46631
I20260619 14:46:54.502192 20361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46631 every 8 connection(s)
I20260619 14:46:54.503140 20234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:54.508123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 20234
I20260619 14:46:54.508456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:46:54.508869 392 external_mini_cluster.cc:1428] Setting key 43de3c2b23c09b96321f7afd6a797b11
I20260619 14:46:54.511171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32987
--builtin_ntp_servers=127.0.98.20:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:46:54.513993 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:54.505447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52041 (local address 127.0.98.62:32987)
0619 14:46:54.505640 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:54.505644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:54.506909 (+ 1265us) server_negotiation.cc:407] Connection header received
0619 14:46:54.508519 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:54.508524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:54.508595 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:54.508669 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:54.509807 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.510447 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.511925 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.512173 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.512291 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:54.513461 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:54.513465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:54.513468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:54.513488 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:54.513507 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:54.513522 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:54.513603 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:54.513776 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 14:46:54.513842 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":95}
I20260619 14:46:54.514946 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:54.505907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32987 (local address 127.0.98.1:52041)
0619 14:46:54.506637 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:54.506682 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:54.508248 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:54.508858 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:54.508866 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:54.508978 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:54.509642 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:54.509656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.510591 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.510596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:54.511785 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:54.511800 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.512338 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.512342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:54.512516 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:54.513175 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:46:54.513177 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:54.513217 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:46:54.513286 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:54.513617 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:54.513628 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:54.513636 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:46:54.513773 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 14:46:54.514085 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":132,"threads_started":1}
I20260619 14:46:54.516448 20362 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32987
I20260619 14:46:54.516760 20362 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:54.517324 20362 heartbeater.cc:511] Master 127.0.98.62:32987 requested a full tablet report, sending...
I20260619 14:46:54.519093 20174 ts_manager.cc:194] Registered new tserver with Master: f81e494bc993463eae313ea8ab3e8471 (127.0.98.1:46631)
I20260619 14:46:54.520562 20174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:52041
W20260619 14:46:54.637884 20365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:54.638371 20365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:54.638456 20365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:54.643359 20365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:54.643491 20365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:54.643608 20365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:54.649492 20365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:54.650729 20365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:54.652484 20365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:54.663790 20370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:54.663997 20365 server_base.cc:1034] running on GCE node
W20260619 14:46:54.663790 20371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:54.663791 20373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:54.664597 20365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:54.665279 20365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:54.666489 20365 hybrid_clock.cc:648] HybridClock initialized: now 1781880414666450 us; error 51 us; skew 500 ppm
I20260619 14:46:54.668892 20365 webserver.cc:460] Webserver started at http://127.0.98.2:41737/ using document root <none> and password file <none>
I20260619 14:46:54.669678 20365 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:54.669757 20365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:54.669976 20365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:54.672271 20365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-1/data/instance:
uuid: "ecd323b4ff47466cb8d52d40f2811f66"
format_stamp: "Formatted at 2026-06-19 14:46:54 on dist-test-slave-cr22"
server_key: "fd16e1a3ba1dbfb89fd1d6c1e6009fcf"
server_key_iv: "2702975b6b06aff6155cffa0bb18f72a"
server_key_version: "encryptionkey@0"
I20260619 14:46:54.672861 20365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ecd323b4ff47466cb8d52d40f2811f66"
format_stamp: "Formatted at 2026-06-19 14:46:54 on dist-test-slave-cr22"
server_key: "fd16e1a3ba1dbfb89fd1d6c1e6009fcf"
server_key_iv: "2702975b6b06aff6155cffa0bb18f72a"
server_key_version: "encryptionkey@0"
I20260619 14:46:54.676959 20365 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:46:54.679925 20379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:54.681746 20365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:54.681855 20365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-1/wal
uuid: "ecd323b4ff47466cb8d52d40f2811f66"
format_stamp: "Formatted at 2026-06-19 14:46:54 on dist-test-slave-cr22"
server_key: "fd16e1a3ba1dbfb89fd1d6c1e6009fcf"
server_key_iv: "2702975b6b06aff6155cffa0bb18f72a"
server_key_version: "encryptionkey@0"
I20260619 14:46:54.682009 20365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:54.700526 20365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:54.701288 20365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:54.701495 20365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:54.702107 20365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:54.703070 20365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:54.703119 20365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:54.703161 20365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:54.703176 20365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:54.713059 20365 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40303
I20260619 14:46:54.713104 20492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40303 every 8 connection(s)
I20260619 14:46:54.714002 20365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:54.718844 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 20365
I20260619 14:46:54.718989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:46:54.719297 392 external_mini_cluster.cc:1428] Setting key d73ccb8990379592b5fbfcebcc2ab5e5
I20260619 14:46:54.721764 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32987
--builtin_ntp_servers=127.0.98.20:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:46:54.724852 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:54.716035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51613 (local address 127.0.98.62:32987)
0619 14:46:54.716176 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:54.716179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:54.717190 (+ 1011us) server_negotiation.cc:407] Connection header received
0619 14:46:54.718590 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:54.718593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:54.718641 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:54.718722 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:54.719978 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.720772 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.722648 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.722868 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.722962 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:54.724269 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:54.724276 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:54.724282 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:54.724301 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:54.724326 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:54.724338 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:54.724423 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:54.724570 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 14:46:54.724643 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:46:54.726007 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:54.716364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32987 (local address 127.0.98.2:51613)
0619 14:46:54.717022 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:54.717062 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:54.718377 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:54.718940 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:54.718951 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:54.719067 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:54.719800 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:54.719817 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.720934 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.720937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:54.722497 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:54.722512 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.723033 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.723040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:54.723288 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:54.723995 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:46:54.723997 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:54.724045 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:46:54.724110 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:54.724430 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:54.724436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:54.724441 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:46:54.724539 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 14:46:54.724799 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
I20260619 14:46:54.727360 20493 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32987
I20260619 14:46:54.727627 20493 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:54.728243 20493 heartbeater.cc:511] Master 127.0.98.62:32987 requested a full tablet report, sending...
I20260619 14:46:54.729671 20174 ts_manager.cc:194] Registered new tserver with Master: ecd323b4ff47466cb8d52d40f2811f66 (127.0.98.2:40303)
I20260619 14:46:54.730621 20174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:51613
W20260619 14:46:54.838070 20496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:54.838351 20496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:54.838392 20496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:54.842680 20496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:54.842751 20496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:54.842810 20496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:54.847877 20496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:54.849026 20496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:54.850457 20496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:54.860417 20504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:54.860457 20501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:54.860527 20502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:54.860658 20496 server_base.cc:1034] running on GCE node
I20260619 14:46:54.861122 20496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:54.861742 20496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:54.862973 20496 hybrid_clock.cc:648] HybridClock initialized: now 1781880414862908 us; error 90 us; skew 500 ppm
I20260619 14:46:54.865044 20496 webserver.cc:460] Webserver started at http://127.0.98.3:42439/ using document root <none> and password file <none>
I20260619 14:46:54.865689 20496 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:54.865783 20496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:54.866015 20496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:54.868234 20496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-2/data/instance:
uuid: "451174f738b44ae3b41df3b4dbe3de63"
format_stamp: "Formatted at 2026-06-19 14:46:54 on dist-test-slave-cr22"
server_key: "de231c61eadd6508a60ee3d875b610a2"
server_key_iv: "69b5e220240aea19d5f63787494634d7"
server_key_version: "encryptionkey@0"
I20260619 14:46:54.868815 20496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-2/wal/instance:
uuid: "451174f738b44ae3b41df3b4dbe3de63"
format_stamp: "Formatted at 2026-06-19 14:46:54 on dist-test-slave-cr22"
server_key: "de231c61eadd6508a60ee3d875b610a2"
server_key_iv: "69b5e220240aea19d5f63787494634d7"
server_key_version: "encryptionkey@0"
I20260619 14:46:54.872897 20496 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:46:54.875560 20510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:54.876937 20496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:54.877089 20496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-2/wal
uuid: "451174f738b44ae3b41df3b4dbe3de63"
format_stamp: "Formatted at 2026-06-19 14:46:54 on dist-test-slave-cr22"
server_key: "de231c61eadd6508a60ee3d875b610a2"
server_key_iv: "69b5e220240aea19d5f63787494634d7"
server_key_version: "encryptionkey@0"
I20260619 14:46:54.877215 20496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:54.890632 20496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:54.891404 20496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:54.891717 20496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:54.892377 20496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:54.893378 20496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:54.893447 20496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:54.893514 20496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:54.893563 20496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:54.904027 20496 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36861
I20260619 14:46:54.904055 20623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36861 every 8 connection(s)
I20260619 14:46:54.904999 20496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:54.909163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 20496
I20260619 14:46:54.909281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:46:54.909555 392 external_mini_cluster.cc:1428] Setting key f409364bc0f74f228c24c9f25f9c3a88
I20260619 14:46:54.916451 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:54.907334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35525 (local address 127.0.98.62:32987)
0619 14:46:54.907523 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:54.907528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:54.908887 (+ 1359us) server_negotiation.cc:407] Connection header received
0619 14:46:54.910728 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:54.910732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:54.910795 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:54.910882 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:54.912167 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.912768 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.914090 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.914341 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.914472 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:54.915851 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:54.915855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:54.915858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:54.915880 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:54.915901 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:54.915911 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:54.915998 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:54.916214 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 14:46:54.916284 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:46:54.917636 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:54.907921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32987 (local address 127.0.98.3:35525)
0619 14:46:54.908709 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:54.908761 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:54.910423 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:54.911080 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:54.911088 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:54.911222 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:54.912002 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:54.912020 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.912921 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.912925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:54.913938 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:54.913949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.914521 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.914525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:54.914727 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:54.915551 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:46:54.915553 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:54.915598 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:46:54.915673 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:54.916026 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:54.916038 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:54.916046 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:46:54.916192 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 14:46:54.916513 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":119,"threads_started":1}
I20260619 14:46:54.919421 20624 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32987
I20260619 14:46:54.919775 20624 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:54.920403 20624 heartbeater.cc:511] Master 127.0.98.62:32987 requested a full tablet report, sending...
I20260619 14:46:54.921805 20174 ts_manager.cc:194] Registered new tserver with Master: 451174f738b44ae3b41df3b4dbe3de63 (127.0.98.3:36861)
I20260619 14:46:54.922528 20174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:35525
I20260619 14:46:54.924309 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:46:54.930364 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:54.926344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43786 (local address 127.0.98.62:32987)
0619 14:46:54.926480 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:54.926484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:54.926642 (+ 158us) server_negotiation.cc:407] Connection header received
0619 14:46:54.926727 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:54.926729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:54.926769 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:54.926841 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:54.927197 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.927849 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.928717 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.928950 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.929039 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:54.929875 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:54.929878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:54.929880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:54.929892 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:54.929910 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:54.929918 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:54.929984 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:54.930201 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 14:46:54.930254 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:46:54.933857 20173 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:46:54.937284 20173 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:46:54.952739 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:54.946753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33916 (local address 127.0.98.2:40303)
0619 14:46:54.947233 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:54.947247 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:54.947642 (+ 395us) server_negotiation.cc:407] Connection header received
0619 14:46:54.947942 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:54.947948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:54.948148 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:54.948314 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:54.948781 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.949977 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.951304 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.952034 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.952160 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:54.952353 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:54.952458 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 14:46:54.952564 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":145,"threads_started":1}
I20260619 14:46:54.952745 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:54.946589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40303 (local address 127.0.0.1:33916)
0619 14:46:54.947500 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:54.947550 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:54.947772 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:54.948491 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:54.948499 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:54.948523 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:54.948660 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:54.948665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.950115 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.950119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:54.951170 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:54.951180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.952161 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.952164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:54.952278 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:54.952318 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 14:46:54.952431 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 14:46:54.952562 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":111,"spinlock_wait_cycles":4352,"thread_start_us":94,"threads_started":1}
I20260619 14:46:54.953541 20640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:54.948091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36861 (local address 127.0.0.1:43708)
0619 14:46:54.948340 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:54.948353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:54.948430 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:54.948912 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:54.948915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:54.948933 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:54.949006 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:54.949012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.950047 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.950051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:54.951761 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:54.951773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.953056 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.953059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:54.953206 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:54.953230 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:46:54.953304 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:46:54.953387 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":165,"thread_start_us":95,"threads_started":1}
I20260619 14:46:54.953550 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:54.946589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46631 (local address 127.0.0.1:45732)
0619 14:46:54.947522 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:54.947569 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:54.947769 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:54.948517 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:54.948520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:54.948538 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:54.948661 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:54.948667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.950115 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.950119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:54.951774 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:54.951780 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.953065 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.953069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:54.953181 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:54.953216 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 14:46:54.953327 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 14:46:54.953403 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":900,"spinlock_wait_cycles":10496,"thread_start_us":99,"threads_started":1}
I20260619 14:46:54.954030 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:54.946865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45732 (local address 127.0.98.1:46631)
0619 14:46:54.947338 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:54.947349 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:54.947648 (+ 299us) server_negotiation.cc:407] Connection header received
0619 14:46:54.947963 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:54.947969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:54.948199 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:54.948329 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:54.948822 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.949994 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.951925 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.952706 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.952813 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:54.952998 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:54.953758 (+ 760us) server_negotiation.cc:299] Negotiation successful
0619 14:46:54.953873 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":137,"threads_started":1}
I20260619 14:46:54.954959 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:54.947732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43708 (local address 127.0.98.3:36861)
0619 14:46:54.948464 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:54.948476 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:54.948499 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:46:54.948566 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:54.948570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:54.948766 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:54.948935 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:54.949128 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.949930 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.951899 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.952699 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.952815 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:54.952998 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:54.954754 (+ 1756us) server_negotiation.cc:299] Negotiation successful
0619 14:46:54.954836 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":139,"threads_started":1}
I20260619 14:46:54.956440 20296 tablet_service.cc:1467] Processing CreateTablet for tablet f683cdab0051474e867293fbe4f7fa27 (DEFAULT_TABLE table=test-table [id=1516dd268e514c9ba609887e0ca183f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:54.956440 20427 tablet_service.cc:1467] Processing CreateTablet for tablet f683cdab0051474e867293fbe4f7fa27 (DEFAULT_TABLE table=test-table [id=1516dd268e514c9ba609887e0ca183f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:54.957567 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f683cdab0051474e867293fbe4f7fa27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:54.957567 20427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f683cdab0051474e867293fbe4f7fa27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:54.957208 20558 tablet_service.cc:1467] Processing CreateTablet for tablet f683cdab0051474e867293fbe4f7fa27 (DEFAULT_TABLE table=test-table [id=1516dd268e514c9ba609887e0ca183f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:54.958186 20558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f683cdab0051474e867293fbe4f7fa27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:54.964128 20641 tablet_bootstrap.cc:492] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63: Bootstrap starting.
I20260619 14:46:54.965353 20642 tablet_bootstrap.cc:492] T f683cdab0051474e867293fbe4f7fa27 P f81e494bc993463eae313ea8ab3e8471: Bootstrap starting.
I20260619 14:46:54.965498 20643 tablet_bootstrap.cc:492] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66: Bootstrap starting.
I20260619 14:46:54.966503 20641 tablet_bootstrap.cc:654] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:54.967265 20641 log.cc:824] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:54.967567 20642 tablet_bootstrap.cc:654] T f683cdab0051474e867293fbe4f7fa27 P f81e494bc993463eae313ea8ab3e8471: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:54.967567 20643 tablet_bootstrap.cc:654] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:54.968386 20642 log.cc:824] T f683cdab0051474e867293fbe4f7fa27 P f81e494bc993463eae313ea8ab3e8471: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:54.968386 20643 log.cc:824] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:54.969841 20641 tablet_bootstrap.cc:492] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63: No bootstrap required, opened a new log
I20260619 14:46:54.970038 20641 ts_tablet_manager.cc:1397] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:46:54.970268 20642 tablet_bootstrap.cc:492] T f683cdab0051474e867293fbe4f7fa27 P f81e494bc993463eae313ea8ab3e8471: No bootstrap required, opened a new log
I20260619 14:46:54.970445 20642 ts_tablet_manager.cc:1397] T f683cdab0051474e867293fbe4f7fa27 P f81e494bc993463eae313ea8ab3e8471: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260619 14:46:54.970726 20643 tablet_bootstrap.cc:492] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66: No bootstrap required, opened a new log
I20260619 14:46:54.970919 20643 ts_tablet_manager.cc:1397] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260619 14:46:54.973702 20641 raft_consensus.cc:348] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd323b4ff47466cb8d52d40f2811f66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40303 } } peers { permanent_uuid: "f81e494bc993463eae313ea8ab3e8471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } } peers { permanent_uuid: "451174f738b44ae3b41df3b4dbe3de63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36861 } }
I20260619 14:46:54.973956 20641 raft_consensus.cc:374] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:54.974035 20641 raft_consensus.cc:729] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451174f738b44ae3b41df3b4dbe3de63, State: Initialized, Role: FOLLOWER
I20260619 14:46:54.974501 20641 consensus_queue.cc:260] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd323b4ff47466cb8d52d40f2811f66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40303 } } peers { permanent_uuid: "f81e494bc993463eae313ea8ab3e8471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } } peers { permanent_uuid: "451174f738b44ae3b41df3b4dbe3de63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36861 } }
I20260619 14:46:54.974787 20642 raft_consensus.cc:348] T f683cdab0051474e867293fbe4f7fa27 P f81e494bc993463eae313ea8ab3e8471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd323b4ff47466cb8d52d40f2811f66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40303 } } peers { permanent_uuid: "f81e494bc993463eae313ea8ab3e8471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } } peers { permanent_uuid: "451174f738b44ae3b41df3b4dbe3de63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36861 } }
I20260619 14:46:54.974951 20643 raft_consensus.cc:348] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd323b4ff47466cb8d52d40f2811f66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40303 } } peers { permanent_uuid: "f81e494bc993463eae313ea8ab3e8471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } } peers { permanent_uuid: "451174f738b44ae3b41df3b4dbe3de63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36861 } }
I20260619 14:46:54.975113 20642 raft_consensus.cc:374] T f683cdab0051474e867293fbe4f7fa27 P f81e494bc993463eae313ea8ab3e8471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:54.975171 20642 raft_consensus.cc:729] T f683cdab0051474e867293fbe4f7fa27 P f81e494bc993463eae313ea8ab3e8471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f81e494bc993463eae313ea8ab3e8471, State: Initialized, Role: FOLLOWER
I20260619 14:46:54.975221 20643 raft_consensus.cc:374] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:54.975291 20643 raft_consensus.cc:729] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecd323b4ff47466cb8d52d40f2811f66, State: Initialized, Role: FOLLOWER
I20260619 14:46:54.975596 20641 ts_tablet_manager.cc:1428] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260619 14:46:54.975644 20642 consensus_queue.cc:260] T f683cdab0051474e867293fbe4f7fa27 P f81e494bc993463eae313ea8ab3e8471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd323b4ff47466cb8d52d40f2811f66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40303 } } peers { permanent_uuid: "f81e494bc993463eae313ea8ab3e8471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } } peers { permanent_uuid: "451174f738b44ae3b41df3b4dbe3de63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36861 } }
I20260619 14:46:54.975672 20643 consensus_queue.cc:260] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd323b4ff47466cb8d52d40f2811f66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40303 } } peers { permanent_uuid: "f81e494bc993463eae313ea8ab3e8471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } } peers { permanent_uuid: "451174f738b44ae3b41df3b4dbe3de63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36861 } }
I20260619 14:46:54.976050 20624 heartbeater.cc:503] Master 127.0.98.62:32987 was elected leader, sending a full tablet report...
I20260619 14:46:54.976310 20362 heartbeater.cc:503] Master 127.0.98.62:32987 was elected leader, sending a full tablet report...
I20260619 14:46:54.976550 20642 ts_tablet_manager.cc:1428] T f683cdab0051474e867293fbe4f7fa27 P f81e494bc993463eae313ea8ab3e8471: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:46:54.977500 20493 heartbeater.cc:503] Master 127.0.98.62:32987 was elected leader, sending a full tablet report...
I20260619 14:46:54.977629 20643 ts_tablet_manager.cc:1428] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
W20260619 14:46:55.004634 20363 tablet.cc:2369] T f683cdab0051474e867293fbe4f7fa27 P f81e494bc993463eae313ea8ab3e8471: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:46:55.144731 20649 raft_consensus.cc:482] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:55.144948 20649 raft_consensus.cc:504] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd323b4ff47466cb8d52d40f2811f66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40303 } } peers { permanent_uuid: "f81e494bc993463eae313ea8ab3e8471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } } peers { permanent_uuid: "451174f738b44ae3b41df3b4dbe3de63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36861 } }
I20260619 14:46:55.146219 20649 leader_election.cc:288] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f81e494bc993463eae313ea8ab3e8471 (127.0.98.1:46631), 451174f738b44ae3b41df3b4dbe3de63 (127.0.98.3:36861)
I20260619 14:46:55.150629 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:55.146501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46631 (local address 127.0.98.2:60509)
0619 14:46:55.146671 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:55.146690 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:55.146785 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:55.147196 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:55.147198 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:55.147216 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:55.147351 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:55.147361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.148399 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.148403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:55.149356 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:55.149366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.150107 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.150111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:55.150231 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:55.150274 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 14:46:55.150358 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 14:46:55.150492 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":54}
I20260619 14:46:55.150665 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:55.146619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60509 (local address 127.0.98.1:46631)
0619 14:46:55.146771 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:55.146776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:55.146796 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:46:55.146908 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:55.146911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:55.146970 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:55.147104 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:55.147624 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.148261 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.149496 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.149947 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.150034 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:55.150087 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:55.150439 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 14:46:55.150525 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:46:55.150907 20650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:55.146779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36861 (local address 127.0.98.2:34973)
0619 14:46:55.147160 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:55.147174 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:55.147335 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:55.147614 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:55.147617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:55.147633 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:55.147752 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:55.147758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.148629 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.148631 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:55.149470 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:55.149481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.150456 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.150460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:55.150554 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:55.150578 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:46:55.150665 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:46:55.150753 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":167,"threads_started":1}
I20260619 14:46:55.151270 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:55.146909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34973 (local address 127.0.98.3:36861)
0619 14:46:55.147070 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:55.147075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:55.147336 (+ 261us) server_negotiation.cc:407] Connection header received
0619 14:46:55.147425 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:55.147452 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:55.147517 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:55.147589 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:55.147914 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.148502 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.149628 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.150188 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.150334 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:55.150400 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:55.150717 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 14:46:55.150805 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 14:46:55.151599 20316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f683cdab0051474e867293fbe4f7fa27" candidate_uuid: "ecd323b4ff47466cb8d52d40f2811f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f81e494bc993463eae313ea8ab3e8471" is_pre_election: true
I20260619 14:46:55.151903 20578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f683cdab0051474e867293fbe4f7fa27" candidate_uuid: "ecd323b4ff47466cb8d52d40f2811f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451174f738b44ae3b41df3b4dbe3de63" is_pre_election: true
I20260619 14:46:55.151965 20316 raft_consensus.cc:2454] T f683cdab0051474e867293fbe4f7fa27 P f81e494bc993463eae313ea8ab3e8471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecd323b4ff47466cb8d52d40f2811f66 in term 0.
I20260619 14:46:55.152180 20578 raft_consensus.cc:2454] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecd323b4ff47466cb8d52d40f2811f66 in term 0.
I20260619 14:46:55.152506 20383 leader_election.cc:302] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ecd323b4ff47466cb8d52d40f2811f66, f81e494bc993463eae313ea8ab3e8471; no voters:
I20260619 14:46:55.152851 20649 raft_consensus.cc:2781] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:46:55.152935 20649 raft_consensus.cc:482] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:46:55.152966 20649 raft_consensus.cc:3037] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:55.154138 20649 raft_consensus.cc:504] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd323b4ff47466cb8d52d40f2811f66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40303 } } peers { permanent_uuid: "f81e494bc993463eae313ea8ab3e8471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } } peers { permanent_uuid: "451174f738b44ae3b41df3b4dbe3de63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36861 } }
I20260619 14:46:55.154505 20649 leader_election.cc:288] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [CANDIDATE]: Term 1 election: Requested vote from peers f81e494bc993463eae313ea8ab3e8471 (127.0.98.1:46631), 451174f738b44ae3b41df3b4dbe3de63 (127.0.98.3:36861)
I20260619 14:46:55.155005 20316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f683cdab0051474e867293fbe4f7fa27" candidate_uuid: "ecd323b4ff47466cb8d52d40f2811f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f81e494bc993463eae313ea8ab3e8471"
I20260619 14:46:55.155117 20578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f683cdab0051474e867293fbe4f7fa27" candidate_uuid: "ecd323b4ff47466cb8d52d40f2811f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451174f738b44ae3b41df3b4dbe3de63"
I20260619 14:46:55.155162 20316 raft_consensus.cc:3037] T f683cdab0051474e867293fbe4f7fa27 P f81e494bc993463eae313ea8ab3e8471 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:55.155280 20578 raft_consensus.cc:3037] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63 [term 0 FOLLOWER]: Advancing to term 1
W20260619 14:46:55.156225 20625 tablet.cc:2369] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:46:55.156412 20578 raft_consensus.cc:2454] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecd323b4ff47466cb8d52d40f2811f66 in term 1.
I20260619 14:46:55.156605 20316 raft_consensus.cc:2454] T f683cdab0051474e867293fbe4f7fa27 P f81e494bc993463eae313ea8ab3e8471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecd323b4ff47466cb8d52d40f2811f66 in term 1.
I20260619 14:46:55.156826 20383 leader_election.cc:302] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451174f738b44ae3b41df3b4dbe3de63, ecd323b4ff47466cb8d52d40f2811f66; no voters:
I20260619 14:46:55.157018 20649 raft_consensus.cc:2781] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:55.157296 20649 raft_consensus.cc:686] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [term 1 LEADER]: Becoming Leader. State: Replica: ecd323b4ff47466cb8d52d40f2811f66, State: Running, Role: LEADER
I20260619 14:46:55.157657 20649 consensus_queue.cc:237] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd323b4ff47466cb8d52d40f2811f66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40303 } } peers { permanent_uuid: "f81e494bc993463eae313ea8ab3e8471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } } peers { permanent_uuid: "451174f738b44ae3b41df3b4dbe3de63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36861 } }
I20260619 14:46:55.161195 20174 catalog_manager.cc:5495] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecd323b4ff47466cb8d52d40f2811f66 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ecd323b4ff47466cb8d52d40f2811f66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd323b4ff47466cb8d52d40f2811f66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f81e494bc993463eae313ea8ab3e8471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "451174f738b44ae3b41df3b4dbe3de63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36861 } health_report { overall_health: UNKNOWN } } }
I20260619 14:46:55.190590 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:55.186771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.0.98.2:40303)
0619 14:46:55.186910 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:55.186914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:55.186926 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:46:55.187000 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:55.187003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:55.187058 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:55.187203 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:55.187421 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.187988 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.189034 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.189270 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.189356 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:55.189656 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:55.190152 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:55.190265 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:55.190395 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 14:46:55.190446 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:46:55.197094 20578 raft_consensus.cc:1261] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63 [term 1 FOLLOWER]: Refusing update from remote peer ecd323b4ff47466cb8d52d40f2811f66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:55.197227 20316 raft_consensus.cc:1261] T f683cdab0051474e867293fbe4f7fa27 P f81e494bc993463eae313ea8ab3e8471 [term 1 FOLLOWER]: Refusing update from remote peer ecd323b4ff47466cb8d52d40f2811f66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:55.197947 20649 consensus_queue.cc:1035] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "451174f738b44ae3b41df3b4dbe3de63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:55.198395 20651 consensus_queue.cc:1035] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f81e494bc993463eae313ea8ab3e8471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:55.206652 20654 mvcc.cc:205] Tried to move back new op lower bound from 7298582180638494720 to 7298582180488687616. Current Snapshot: MvccSnapshot[applied={T|T < 7298582180638494720}]
I20260619 14:46:55.209463 20656 mvcc.cc:205] Tried to move back new op lower bound from 7298582180638494720 to 7298582180488687616. Current Snapshot: MvccSnapshot[applied={T|T < 7298582180638494720}]
I20260619 14:46:55.211207 20655 mvcc.cc:205] Tried to move back new op lower bound from 7298582180638494720 to 7298582180488687616. Current Snapshot: MvccSnapshot[applied={T|T < 7298582180638494720 or (T in {7298582180638494720})}]
I20260619 14:46:55.216056 20172 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:55.216259 20172 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:55.218731 20172 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d1ff9b451bf4381a71cdb60a7563ef6: Sending DeleteTablet for 3 replicas of tablet f683cdab0051474e867293fbe4f7fa27
I20260619 14:46:55.219730 20296 tablet_service.cc:1514] Processing DeleteTablet for tablet f683cdab0051474e867293fbe4f7fa27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:55 UTC) from {username='slave'} at 127.0.0.1:45732
I20260619 14:46:55.219908 20427 tablet_service.cc:1514] Processing DeleteTablet for tablet f683cdab0051474e867293fbe4f7fa27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:55 UTC) from {username='slave'} at 127.0.0.1:33916
I20260619 14:46:55.220249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 20234
I20260619 14:46:55.220309 20558 tablet_service.cc:1514] Processing DeleteTablet for tablet f683cdab0051474e867293fbe4f7fa27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:55 UTC) from {username='slave'} at 127.0.0.1:43708
I20260619 14:46:55.222266 20666 tablet_replica.cc:330] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63: stopping tablet replica
I20260619 14:46:55.222611 20666 raft_consensus.cc:2229] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:55.222956 20666 raft_consensus.cc:2258] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:55.224413 20666 ts_tablet_manager.cc:1905] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:55.226660 20666 ts_tablet_manager.cc:1918] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:55.226778 20666 log.cc:1196] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63: Deleting WAL directory at /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880155260304-392-0/minicluster-data/ts-2/wal/wals/f683cdab0051474e867293fbe4f7fa27
I20260619 14:46:55.227114 20666 ts_tablet_manager.cc:1939] T f683cdab0051474e867293fbe4f7fa27 P 451174f738b44ae3b41df3b4dbe3de63: Deleting consensus metadata
I20260619 14:46:55.228787 20161 catalog_manager.cc:4841] TS 451174f738b44ae3b41df3b4dbe3de63 (127.0.98.3:36861): tablet f683cdab0051474e867293fbe4f7fa27 (table test-table [id=1516dd268e514c9ba609887e0ca183f5]) successfully deleted
I20260619 14:46:55.231654 20665 tablet_replica.cc:330] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66: stopping tablet replica
I20260619 14:46:55.232228 20665 raft_consensus.cc:2229] T f683cdab0051474e867293fbe4f7fa27 P ecd323b4ff47466cb8d52d40f2811f66 [term 1 LEADER]: Raft consensus shutting down.
W20260619 14:46:55.232247 20161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46631 (error 108)
I20260619 14:46:55.232470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 20365
I20260619 14:46:55.234341 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:55.234051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46631 (local address 127.0.0.1:45740)
0619 14:46:55.234190 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:55.234264 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 14:46:55.234493 20161 catalog_manager.cc:4584] TS f81e494bc993463eae313ea8ab3e8471 (127.0.98.1:46631): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f683cdab0051474e867293fbe4f7fa27: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46631: connect: Connection refused (error 111)
I20260619 14:46:55.238772 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 20496
I20260619 14:46:55.240137 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:55.239053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40303 (local address 127.0.0.1:33934)
0619 14:46:55.240005 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:55.240058 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":861}
I20260619 14:46:55.245250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 20143
2026-06-19T14:46:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1249 ms)
[----------] 6 tests from EncryptionPolicyTest (10886 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:46:55 dist-test-slave-cr22 krb5kdc[20670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:46:55 dist-test-slave-cr22 krb5kdc[20670](info): set up 2 sockets
Jun 19 14:46:55 dist-test-slave-cr22 krb5kdc[20670](info): commencing operation
krb5kdc: starting...
W20260619 14:46:56.319836 392 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:46:56 dist-test-slave-cr22 krb5kdc[20670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:46:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:46:56.459002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32869
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:56.585489 20683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:56.585786 20683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:56.585835 20683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:56.590092 20683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:56.590176 20683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:56.590196 20683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:56.590214 20683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:56.590229 20683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:56.596058 20683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32869
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:56.597183 20683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:56.598214 20683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:56.606946 20689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:56.606976 20691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:56.606976 20688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:56.607846 20683 server_base.cc:1034] running on GCE node
I20260619 14:46:56.608460 20683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:56.609653 20683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:56.610855 20683 hybrid_clock.cc:648] HybridClock initialized: now 1781880416610824 us; error 46 us; skew 500 ppm
Jun 19 14:46:56 dist-test-slave-cr22 krb5kdc[20670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:56.613884 20683 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:46:56.614866 20683 webserver.cc:460] Webserver started at http://127.0.0.1:36865/ using document root <none> and password file <none>
I20260619 14:46:56.615614 20683 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:56.615716 20683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:56.615968 20683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:56.618201 20683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "8d6210db374a4010a559dbebe8f6a834"
format_stamp: "Formatted at 2026-06-19 14:46:56 on dist-test-slave-cr22"
server_key: "e3d2e987040da7f4c0736443bba3d813"
server_key_iv: "6b9753ba1efb8312c63149e2b1323c27"
server_key_version: "encryptionkey@0"
I20260619 14:46:56.618819 20683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d6210db374a4010a559dbebe8f6a834"
format_stamp: "Formatted at 2026-06-19 14:46:56 on dist-test-slave-cr22"
server_key: "e3d2e987040da7f4c0736443bba3d813"
server_key_iv: "6b9753ba1efb8312c63149e2b1323c27"
server_key_version: "encryptionkey@0"
I20260619 14:46:56.622980 20683 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:46:56.625962 20698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:56.627316 20683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:46:56.627527 20683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "8d6210db374a4010a559dbebe8f6a834"
format_stamp: "Formatted at 2026-06-19 14:46:56 on dist-test-slave-cr22"
server_key: "e3d2e987040da7f4c0736443bba3d813"
server_key_iv: "6b9753ba1efb8312c63149e2b1323c27"
server_key_version: "encryptionkey@0"
I20260619 14:46:56.627663 20683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:56.645315 20683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:56.648869 20683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:56.649085 20683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:56.656270 20683 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32869
I20260619 14:46:56.656278 20750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32869 every 8 connection(s)
I20260619 14:46:56.657383 20683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:56.660336 20751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:56.666115 20751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834: Bootstrap starting.
I20260619 14:46:56.666851 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 20683
I20260619 14:46:56.667012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:56.667320 392 external_mini_cluster.cc:1428] Setting key c9f8c3ad2e278ddeea594e699189f239
I20260619 14:46:56.669305 20751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:56.670090 20751 log.cc:824] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:56.672703 20751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834: No bootstrap required, opened a new log
I20260619 14:46:56.676433 20751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6210db374a4010a559dbebe8f6a834" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32869 } }
I20260619 14:46:56.676708 20751 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:56.676798 20751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6210db374a4010a559dbebe8f6a834, State: Initialized, Role: FOLLOWER
I20260619 14:46:56.677407 20751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6210db374a4010a559dbebe8f6a834" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32869 } }
I20260619 14:46:56.677595 20751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:56.677664 20751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:56.677790 20751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 14:46:56 dist-test-slave-cr22 krb5kdc[20670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:46:56.678866 20751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6210db374a4010a559dbebe8f6a834" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32869 } }
I20260619 14:46:56.679220 20751 leader_election.cc:302] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d6210db374a4010a559dbebe8f6a834; no voters:
I20260619 14:46:56.679642 20751 leader_election.cc:288] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:56.679733 20756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:56.679921 20756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834 [term 1 LEADER]: Becoming Leader. State: Replica: 8d6210db374a4010a559dbebe8f6a834, State: Running, Role: LEADER
I20260619 14:46:56.680306 20756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6210db374a4010a559dbebe8f6a834" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32869 } }
I20260619 14:46:56.681002 20751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:56.681953 20757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d6210db374a4010a559dbebe8f6a834" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6210db374a4010a559dbebe8f6a834" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32869 } } }
I20260619 14:46:56.682091 20757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:56.682299 20757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6210db374a4010a559dbebe8f6a834. Latest consensus state: current_term: 1 leader_uuid: "8d6210db374a4010a559dbebe8f6a834" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6210db374a4010a559dbebe8f6a834" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32869 } } }
I20260619 14:46:56.682367 20757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:56.683292 20765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:56.685508 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:56.668734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38829 (local address 127.0.0.1:32869)
0619 14:46:56.669378 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:56.669399 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:56.669478 (+ 79us) server_negotiation.cc:407] Connection header received
0619 14:46:56.670518 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:56.670550 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:56.671153 (+ 603us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:56.671547 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:56.672625 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:56.674279 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:56.675400 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:56.676054 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:56.676253 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:56.679654 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:56.679660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:56.679676 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:56.679732 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:56.681919 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:56.682582 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:56.682588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:56.682606 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:56.682744 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:56.683159 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:56.683162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:56.683164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:56.683733 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:56.683965 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:56.684318 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 14:46:56.684550 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":150,"threads_started":1}
I20260619 14:46:56.686769 20765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:56.688788 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:46:56.692505 20765 catalog_manager.cc:1347] Generated new cluster ID: e4196d2085bd4dc0a82ae557a5db6f08
I20260619 14:46:56.692571 20765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:56.700812 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:56.691087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53013 (local address 127.0.0.1:32869)
0619 14:46:56.691213 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:56.691218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:56.691510 (+ 292us) server_negotiation.cc:407] Connection header received
0619 14:46:56.691709 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:56.691715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:56.691769 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:56.691828 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:56.692954 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:56.693729 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:56.694928 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:56.695292 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:56.695378 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:56.697160 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:56.697164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:56.697167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:56.697192 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:56.698974 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:56.699528 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:56.699531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:56.699533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:56.699612 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:56.699941 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:56.699944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:56.699951 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:56.700142 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:56.700271 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:56.700553 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 14:46:56.700643 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:46:56.719451 20765 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:56.720439 20765 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:56.740520 20765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d6210db374a4010a559dbebe8f6a834: Generated new TSK 0
I20260619 14:46:56.741508 20765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:46:56.749609 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 20683
W20260619 14:46:56.755633 20668 connection.cc:535] client connection to 127.0.0.1:32869 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32869): Connection reset by peer (error 104)
2026-06-19T14:46:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1502 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:46:56 dist-test-slave-cr22 krb5kdc[20782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:46:56 dist-test-slave-cr22 krb5kdc[20782](info): set up 2 sockets
Jun 19 14:46:56 dist-test-slave-cr22 krb5kdc[20782](info): commencing operation
krb5kdc: starting...
W20260619 14:46:57.863821 392 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:46:57 dist-test-slave-cr22 krb5kdc[20782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:46:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:46:58.004889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40721
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:58.140628 20795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:58.140915 20795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:58.140959 20795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:58.145174 20795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:58.145248 20795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:58.145268 20795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:58.145285 20795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:58.145300 20795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:58.150942 20795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40721
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:58.152139 20795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:58.153178 20795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:58.162760 20803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:58.162973 20801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:58.163061 20795 server_base.cc:1034] running on GCE node
W20260619 14:46:58.162910 20800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:58.164009 20795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:58.165297 20795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:58.166486 20795 hybrid_clock.cc:648] HybridClock initialized: now 1781880418166462 us; error 45 us; skew 500 ppm
Jun 19 14:46:58 dist-test-slave-cr22 krb5kdc[20782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:58.173321 20795 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:46:58.174295 20795 webserver.cc:460] Webserver started at http://127.0.0.1:46591/ using document root <none> and password file <none>
I20260619 14:46:58.174854 20795 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:58.174912 20795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:58.175081 20795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:58.177289 20795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "3fc3a17fba0440af952c98f0d47f0228"
format_stamp: "Formatted at 2026-06-19 14:46:58 on dist-test-slave-cr22"
server_key: "61e06376c3303c931a9faea52d30c2eb"
server_key_iv: "2e14817046569cc7e72bbd4f03a57a10"
server_key_version: "encryptionkey@0"
I20260619 14:46:58.177793 20795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "3fc3a17fba0440af952c98f0d47f0228"
format_stamp: "Formatted at 2026-06-19 14:46:58 on dist-test-slave-cr22"
server_key: "61e06376c3303c931a9faea52d30c2eb"
server_key_iv: "2e14817046569cc7e72bbd4f03a57a10"
server_key_version: "encryptionkey@0"
I20260619 14:46:58.181771 20795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 14:46:58.184340 20810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:58.185714 20795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:46:58.185899 20795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "3fc3a17fba0440af952c98f0d47f0228"
format_stamp: "Formatted at 2026-06-19 14:46:58 on dist-test-slave-cr22"
server_key: "61e06376c3303c931a9faea52d30c2eb"
server_key_iv: "2e14817046569cc7e72bbd4f03a57a10"
server_key_version: "encryptionkey@0"
I20260619 14:46:58.186031 20795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:58.201895 20795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:58.206261 20795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:58.206516 20795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:58.215111 20795 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40721
I20260619 14:46:58.215129 20862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40721 every 8 connection(s)
I20260619 14:46:58.216192 20795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:58.219206 20863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:58.224421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 20795
I20260619 14:46:58.224565 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:58.224859 392 external_mini_cluster.cc:1428] Setting key 4bca495ce91a16b930b5848f071ae8c1
I20260619 14:46:58.225275 20863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228: Bootstrap starting.
I20260619 14:46:58.227905 20863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:58.228652 20863 log.cc:824] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:58.230620 20863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228: No bootstrap required, opened a new log
I20260619 14:46:58.235253 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.226333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37229 (local address 127.0.0.1:40721)
0619 14:46:58.226820 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.226834 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.226871 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:46:58.227645 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.227664 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.228051 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.228389 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:58.228898 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.229848 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.230768 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.231466 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.231678 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.232915 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:58.232922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:58.232940 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:58.233461 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:58.233542 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:58.233679 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:58.233863 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:58.234101 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 14:46:58.234244 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":158,"threads_started":1}
I20260619 14:46:58.235378 20863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc3a17fba0440af952c98f0d47f0228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40721 } }
I20260619 14:46:58.235689 20863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:58.235733 20863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc3a17fba0440af952c98f0d47f0228, State: Initialized, Role: FOLLOWER
I20260619 14:46:58.236207 20863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc3a17fba0440af952c98f0d47f0228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40721 } }
I20260619 14:46:58.236338 20863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:58.236424 20863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:58.236547 20863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:58.237553 20863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc3a17fba0440af952c98f0d47f0228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40721 } }
I20260619 14:46:58.237905 20863 leader_election.cc:302] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fc3a17fba0440af952c98f0d47f0228; no voters:
I20260619 14:46:58.238234 20863 leader_election.cc:288] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:58.238384 20868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:58.238701 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:46:58.238765 20868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228 [term 1 LEADER]: Becoming Leader. State: Replica: 3fc3a17fba0440af952c98f0d47f0228, State: Running, Role: LEADER
I20260619 14:46:58.239104 20868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc3a17fba0440af952c98f0d47f0228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40721 } }
I20260619 14:46:58.239565 20863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:58.240968 20869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fc3a17fba0440af952c98f0d47f0228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc3a17fba0440af952c98f0d47f0228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40721 } } }
I20260619 14:46:58.241096 20869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:58.241436 20870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fc3a17fba0440af952c98f0d47f0228. Latest consensus state: current_term: 1 leader_uuid: "3fc3a17fba0440af952c98f0d47f0228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc3a17fba0440af952c98f0d47f0228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40721 } } }
I20260619 14:46:58.241518 20870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228 [sys.catalog]: This master's current role is: LEADER
W20260619 14:46:58.244776 20888 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:46:58.244923 20888 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:46:58.245033 20883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:58.246026 20883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:58.246294 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.241214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44623 (local address 127.0.0.1:40721)
0619 14:46:58.241341 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.241346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.241682 (+ 336us) server_negotiation.cc:407] Connection header received
0619 14:46:58.241898 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.241901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.241963 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.242049 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:58.242473 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.243221 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.244339 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.244746 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.244865 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.245831 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:58.245836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:58.245838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:58.245859 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:58.245883 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:58.245894 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:58.245994 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:58.246117 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 14:46:58.246148 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:46:58.252873 20883 catalog_manager.cc:1347] Generated new cluster ID: 44b45a5fc3194d679232fff5be37b85a
I20260619 14:46:58.252956 20883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:58.272986 20883 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:58.273970 20883 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:58.281782 20883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fc3a17fba0440af952c98f0d47f0228: Generated new TSK 0
I20260619 14:46:58.282742 20883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:46:58.293500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 20795
W20260619 14:46:58.299088 20780 connection.cc:535] client connection to 127.0.0.1:40721 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40721): Connection reset by peer (error 104)
2026-06-19T14:46:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1543 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:46:58 dist-test-slave-cr22 krb5kdc[20894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:46:58 dist-test-slave-cr22 krb5kdc[20894](info): set up 2 sockets
Jun 19 14:46:58 dist-test-slave-cr22 krb5kdc[20894](info): commencing operation
krb5kdc: starting...
W20260619 14:46:59.428066 392 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:46:59 dist-test-slave-cr22 krb5kdc[20894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:46:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:46:59.567488 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45445
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:59.684782 20907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:59.685062 20907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:59.685105 20907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:59.689273 20907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:59.689345 20907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:59.689364 20907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:59.689381 20907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:59.689397 20907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:59.695330 20907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45445
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:46:59.696563 20907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:59.697620 20907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:46:59.707306 20907 server_base.cc:1034] running on GCE node
W20260619 14:46:59.707266 20912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:59.707360 20913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:59.707266 20915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:59.708125 20907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:59.709259 20907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:59.710431 20907 hybrid_clock.cc:648] HybridClock initialized: now 1781880419710394 us; error 53 us; skew 500 ppm
Jun 19 14:46:59 dist-test-slave-cr22 krb5kdc[20894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:59.713382 20907 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:46:59.714300 20907 webserver.cc:460] Webserver started at http://127.0.0.1:39305/ using document root <none> and password file <none>
I20260619 14:46:59.714880 20907 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:59.714974 20907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:59.715173 20907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:59.717239 20907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "620469de24eb4a8e8073df0bf93cd21a"
format_stamp: "Formatted at 2026-06-19 14:46:59 on dist-test-slave-cr22"
server_key: "3a1a37101e00fac051f05c35b0c8ff0f"
server_key_iv: "0d534aa3ca1e808710d5d42c0fd803b6"
server_key_version: "encryptionkey@0"
I20260619 14:46:59.717741 20907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "620469de24eb4a8e8073df0bf93cd21a"
format_stamp: "Formatted at 2026-06-19 14:46:59 on dist-test-slave-cr22"
server_key: "3a1a37101e00fac051f05c35b0c8ff0f"
server_key_iv: "0d534aa3ca1e808710d5d42c0fd803b6"
server_key_version: "encryptionkey@0"
I20260619 14:46:59.721513 20907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 14:46:59.724275 20922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:59.725682 20907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 14:46:59.725805 20907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "620469de24eb4a8e8073df0bf93cd21a"
format_stamp: "Formatted at 2026-06-19 14:46:59 on dist-test-slave-cr22"
server_key: "3a1a37101e00fac051f05c35b0c8ff0f"
server_key_iv: "0d534aa3ca1e808710d5d42c0fd803b6"
server_key_version: "encryptionkey@0"
I20260619 14:46:59.725904 20907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:59.730475 20907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:59.733732 20907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:59.733880 20907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:59.741180 20907 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45445
I20260619 14:46:59.741220 20974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45445 every 8 connection(s)
I20260619 14:46:59.742121 20907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:59.744026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 20907
I20260619 14:46:59.744166 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:59.744464 392 external_mini_cluster.cc:1428] Setting key 10301d3a342ad0ea7bda761f9ae2d525
I20260619 14:46:59.745098 20975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:59.751611 20978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.745846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39445 (local address 127.0.0.1:45445)
0619 14:46:59.746323 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.746339 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.746375 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:46:59.747363 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.747384 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.747796 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.748089 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:59.749210 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:59.749219 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:59.749240 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:59.749263 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:59.749374 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:59.749520 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:59.749690 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.749994 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.750178 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":113,"threads_started":1}
I20260619 14:46:59.753801 20975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a: Bootstrap starting.
I20260619 14:46:59.756083 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:46:59.756877 20975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:59.757807 20975 log.cc:824] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:59.760747 20975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a: No bootstrap required, opened a new log
I20260619 14:46:59.761102 20978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.758761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58793 (local address 127.0.0.1:45445)
0619 14:46:59.758898 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.758902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.759079 (+ 177us) server_negotiation.cc:407] Connection header received
0619 14:46:59.759411 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.759414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.759512 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.759602 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:59.760388 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:59.760391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:59.760393 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:59.760403 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:59.760429 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:59.760439 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:59.760542 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.760812 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.760908 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:46:59.765116 20975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "620469de24eb4a8e8073df0bf93cd21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45445 } }
I20260619 14:46:59.765419 20975 raft_consensus.cc:374] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:59.765484 20975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620469de24eb4a8e8073df0bf93cd21a, State: Initialized, Role: FOLLOWER
I20260619 14:46:59.765969 20975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "620469de24eb4a8e8073df0bf93cd21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45445 } }
I20260619 14:46:59.766129 20975 raft_consensus.cc:388] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:59.766213 20975 raft_consensus.cc:482] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:59.766310 20975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:59.767616 20975 raft_consensus.cc:504] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "620469de24eb4a8e8073df0bf93cd21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45445 } }
I20260619 14:46:59.768090 20975 leader_election.cc:302] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 620469de24eb4a8e8073df0bf93cd21a; no voters:
I20260619 14:46:59.768518 20975 leader_election.cc:288] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:59.768652 20985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:59.768918 20985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a [term 1 LEADER]: Becoming Leader. State: Replica: 620469de24eb4a8e8073df0bf93cd21a, State: Running, Role: LEADER
I20260619 14:46:59.769445 20985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "620469de24eb4a8e8073df0bf93cd21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45445 } }
I20260619 14:46:59.769835 20975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:59.771337 20987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 620469de24eb4a8e8073df0bf93cd21a. Latest consensus state: current_term: 1 leader_uuid: "620469de24eb4a8e8073df0bf93cd21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "620469de24eb4a8e8073df0bf93cd21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45445 } } }
I20260619 14:46:59.771310 20986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "620469de24eb4a8e8073df0bf93cd21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "620469de24eb4a8e8073df0bf93cd21a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45445 } } }
I20260619 14:46:59.771469 20986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:59.771469 20987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a [sys.catalog]: This master's current role is: LEADER
W20260619 14:46:59.774670 21000 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:46:59.774765 21000 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:46:59.774838 20999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:59.775585 20999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:59.781296 20999 catalog_manager.cc:1347] Generated new cluster ID: c0afec6b5eab4fdda090d6e7a4a9b2f0
I20260619 14:46:59.781394 20999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:59.853138 20999 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:59.853462 20999 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:59.861603 20999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 620469de24eb4a8e8073df0bf93cd21a: Generated new TSK 0
I20260619 14:46:59.862628 20999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:46:59.871275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 20907
2026-06-19T14:46:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1577 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:46:59 dist-test-slave-cr22 krb5kdc[21006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:46:59 dist-test-slave-cr22 krb5kdc[21006](info): set up 2 sockets
Jun 19 14:46:59 dist-test-slave-cr22 krb5kdc[21006](info): commencing operation
krb5kdc: starting...
W20260619 14:47:00.936297 392 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:01 dist-test-slave-cr22 krb5kdc[21006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:47:01.066907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:01.181062 21019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:01.181406 21019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:01.181483 21019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:01.185576 21019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:01.185693 21019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:01.185730 21019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:01.185787 21019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:01.185823 21019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:01.191272 21019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:01.192413 21019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:01.193440 21019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:01.202489 21024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:01.202489 21027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:01.202492 21025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:01.202772 21019 server_base.cc:1034] running on GCE node
I20260619 14:47:01.203496 21019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:01.204725 21019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:01.205938 21019 hybrid_clock.cc:648] HybridClock initialized: now 1781880421205897 us; error 54 us; skew 500 ppm
Jun 19 14:47:01 dist-test-slave-cr22 krb5kdc[21006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:01.208948 21019 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:47:01.210265 21019 webserver.cc:460] Webserver started at http://127.0.0.1:40715/ using document root <none> and password file <none>
I20260619 14:47:01.211153 21019 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:01.211261 21019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:01.211548 21019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:01.214964 21019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "d9652ce6082749da994eabfadaaf7ff9"
format_stamp: "Formatted at 2026-06-19 14:47:01 on dist-test-slave-cr22"
server_key: "ef52d394b6ce9c7f8e46a92fe87b2347"
server_key_iv: "224791ce74d7c30d5daa7b6193b79eef"
server_key_version: "encryptionkey@0"
I20260619 14:47:01.215627 21019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "d9652ce6082749da994eabfadaaf7ff9"
format_stamp: "Formatted at 2026-06-19 14:47:01 on dist-test-slave-cr22"
server_key: "ef52d394b6ce9c7f8e46a92fe87b2347"
server_key_iv: "224791ce74d7c30d5daa7b6193b79eef"
server_key_version: "encryptionkey@0"
I20260619 14:47:01.219836 21019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:47:01.222551 21034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:01.223971 21019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:01.224084 21019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "d9652ce6082749da994eabfadaaf7ff9"
format_stamp: "Formatted at 2026-06-19 14:47:01 on dist-test-slave-cr22"
server_key: "ef52d394b6ce9c7f8e46a92fe87b2347"
server_key_iv: "224791ce74d7c30d5daa7b6193b79eef"
server_key_version: "encryptionkey@0"
I20260619 14:47:01.224237 21019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:01.249378 21019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:01.253098 21019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:01.253541 21019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:01.262430 21019 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41217
I20260619 14:47:01.262456 21086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41217 every 8 connection(s)
I20260619 14:47:01.263589 21019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:01.267225 21087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:01.268142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 21019
I20260619 14:47:01.268335 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:01.268666 392 external_mini_cluster.cc:1428] Setting key c578f9be9ce4b655a46c8305c251096d
I20260619 14:47:01.274989 21087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9: Bootstrap starting.
Jun 19 14:47:01 dist-test-slave-cr22 krb5kdc[21006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:47:01.278065 21087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:01.278847 21087 log.cc:824] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:01.281111 21087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9: No bootstrap required, opened a new log
I20260619 14:47:01.283138 21090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.270127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60949 (local address 127.0.0.1:41217)
0619 14:47:01.270576 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:01.270591 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:01.270628 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:47:01.271360 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:01.271379 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:01.271865 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:01.272203 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:01.273005 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.273975 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.274853 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.275394 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.275608 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:01.278164 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:01.278174 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:01.278196 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:01.278236 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:01.280191 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:01.280646 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:01.280654 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:01.280664 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:01.280769 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:01.281119 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:01.281123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:01.281125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:01.281572 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:01.281788 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:01.282095 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 14:47:01.282394 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":134,"threads_started":1}
I20260619 14:47:01.285089 21087 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9652ce6082749da994eabfadaaf7ff9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41217 } }
I20260619 14:47:01.285344 21087 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:01.285415 21087 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9652ce6082749da994eabfadaaf7ff9, State: Initialized, Role: FOLLOWER
I20260619 14:47:01.285907 21087 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9652ce6082749da994eabfadaaf7ff9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41217 } }
I20260619 14:47:01.286073 21087 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:01.286142 21087 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:01.286291 21087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:01.286338 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:47:01.287237 21087 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9652ce6082749da994eabfadaaf7ff9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41217 } }
I20260619 14:47:01.287614 21087 leader_election.cc:302] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9652ce6082749da994eabfadaaf7ff9; no voters:
I20260619 14:47:01.287966 21087 leader_election.cc:288] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:01.288070 21096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:01.288865 21096 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9 [term 1 LEADER]: Becoming Leader. State: Replica: d9652ce6082749da994eabfadaaf7ff9, State: Running, Role: LEADER
I20260619 14:47:01.289274 21096 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9652ce6082749da994eabfadaaf7ff9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41217 } }
I20260619 14:47:01.289364 21087 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:01.291203 21099 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9652ce6082749da994eabfadaaf7ff9. Latest consensus state: current_term: 1 leader_uuid: "d9652ce6082749da994eabfadaaf7ff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9652ce6082749da994eabfadaaf7ff9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41217 } } }
I20260619 14:47:01.291203 21097 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9652ce6082749da994eabfadaaf7ff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9652ce6082749da994eabfadaaf7ff9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41217 } } }
I20260619 14:47:01.291373 21099 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:01.291373 21097 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9 [sys.catalog]: This master's current role is: LEADER
W20260619 14:47:01.294550 21111 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:47:01.294651 21111 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:47:01.294838 21112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:01.295814 21112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:01.297734 21090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.288644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45689 (local address 127.0.0.1:41217)
0619 14:47:01.288782 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:01.288787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:01.288971 (+ 184us) server_negotiation.cc:407] Connection header received
0619 14:47:01.289167 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:01.289170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:01.289220 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:01.289315 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:01.290391 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.291232 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.292483 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.292731 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.292843 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:01.294263 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:01.294267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:01.294270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:01.294294 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:01.296028 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:01.296458 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:01.296462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:01.296464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:01.296599 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:01.296876 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:01.296881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:01.296883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:01.297067 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:01.297202 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:01.297435 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 14:47:01.297558 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:47:01.307289 21112 catalog_manager.cc:1347] Generated new cluster ID: 92e95e606557407da6edce7421266293
I20260619 14:47:01.307371 21112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:01.317265 21112 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:01.318352 21112 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:01.326623 21112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9652ce6082749da994eabfadaaf7ff9: Generated new TSK 0
I20260619 14:47:01.327405 21112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:47:01.332181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 21019
2026-06-19T14:47:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:01 dist-test-slave-cr22 krb5kdc[21118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:01 dist-test-slave-cr22 krb5kdc[21118](info): set up 2 sockets
Jun 19 14:47:01 dist-test-slave-cr22 krb5kdc[21118](info): commencing operation
krb5kdc: starting...
W20260619 14:47:02.403882 392 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:02 dist-test-slave-cr22 krb5kdc[21118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:47:02.539788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39743
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:02.661253 21131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:02.661518 21131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:02.661561 21131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:02.665622 21131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:02.665684 21131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:02.665704 21131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:02.665720 21131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:02.665735 21131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:02.670996 21131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:02.672204 21131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:02.673228 21131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:02.681959 21137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:02.681975 21136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:02.682017 21131 server_base.cc:1034] running on GCE node
W20260619 14:47:02.681965 21139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:02.682886 21131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:02.684053 21131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:02.685273 21131 hybrid_clock.cc:648] HybridClock initialized: now 1781880422685246 us; error 40 us; skew 500 ppm
Jun 19 14:47:02 dist-test-slave-cr22 krb5kdc[21118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:02.688377 21131 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:47:02.689227 21131 webserver.cc:460] Webserver started at http://127.0.0.1:35243/ using document root <none> and password file <none>
I20260619 14:47:02.689808 21131 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:02.689869 21131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:02.690084 21131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:02.692268 21131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "ca3cdb05969c450da05a6b6e51550731"
format_stamp: "Formatted at 2026-06-19 14:47:02 on dist-test-slave-cr22"
server_key: "d4b5e8b54277e5b1f5f1334652c9f626"
server_key_iv: "02f596c913dbaca65acd8aade60b4429"
server_key_version: "encryptionkey@0"
I20260619 14:47:02.692858 21131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca3cdb05969c450da05a6b6e51550731"
format_stamp: "Formatted at 2026-06-19 14:47:02 on dist-test-slave-cr22"
server_key: "d4b5e8b54277e5b1f5f1334652c9f626"
server_key_iv: "02f596c913dbaca65acd8aade60b4429"
server_key_version: "encryptionkey@0"
I20260619 14:47:02.696873 21131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 14:47:02.699666 21146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:02.700874 21131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:02.701010 21131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "ca3cdb05969c450da05a6b6e51550731"
format_stamp: "Formatted at 2026-06-19 14:47:02 on dist-test-slave-cr22"
server_key: "d4b5e8b54277e5b1f5f1334652c9f626"
server_key_iv: "02f596c913dbaca65acd8aade60b4429"
server_key_version: "encryptionkey@0"
I20260619 14:47:02.701128 21131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:02.724862 21131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:02.728253 21131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:02.728463 21131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:02.735819 21131 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39743
I20260619 14:47:02.735888 21198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39743 every 8 connection(s)
I20260619 14:47:02.737463 21131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:02.740587 21199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:02.746523 21199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731: Bootstrap starting.
I20260619 14:47:02.747753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 21131
I20260619 14:47:02.747918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:02.748277 392 external_mini_cluster.cc:1428] Setting key fe9fc29f685dcf9bdfdb196c78e3dc0c
I20260619 14:47:02.749504 21199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:02.750313 21199 log.cc:824] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:02.752512 21199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731: No bootstrap required, opened a new log
I20260619 14:47:02.756860 21199 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3cdb05969c450da05a6b6e51550731" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39743 } }
I20260619 14:47:02.757112 21199 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:02.757194 21199 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3cdb05969c450da05a6b6e51550731, State: Initialized, Role: FOLLOWER
I20260619 14:47:02.757615 21199 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3cdb05969c450da05a6b6e51550731" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39743 } }
I20260619 14:47:02.757773 21199 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:02.757859 21199 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:02.757977 21199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:02.758020 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:02.749568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37525 (local address 127.0.0.1:39743)
0619 14:47:02.750039 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:02.750056 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:02.750089 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:47:02.750828 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:02.750851 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:02.751259 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:02.751618 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:02.751917 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:02.752917 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:02.754274 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:02.754758 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:02.754887 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:02.755918 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:02.755923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:02.755935 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:02.756344 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:02.756401 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:02.756424 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:02.756505 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:02.756807 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 14:47:02.757024 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":154,"threads_started":1}
I20260619 14:47:02.759150 21199 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3cdb05969c450da05a6b6e51550731" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39743 } }
I20260619 14:47:02.759555 21199 leader_election.cc:302] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca3cdb05969c450da05a6b6e51550731; no voters:
I20260619 14:47:02.759899 21199 leader_election.cc:288] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:02.760001 21204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:02.760176 21204 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731 [term 1 LEADER]: Becoming Leader. State: Replica: ca3cdb05969c450da05a6b6e51550731, State: Running, Role: LEADER
I20260619 14:47:02.760520 21204 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3cdb05969c450da05a6b6e51550731" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39743 } }
I20260619 14:47:02.761142 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:47:02.761158 21199 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:02.762192 21205 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca3cdb05969c450da05a6b6e51550731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3cdb05969c450da05a6b6e51550731" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39743 } } }
I20260619 14:47:02.762315 21205 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:02.762676 21206 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca3cdb05969c450da05a6b6e51550731. Latest consensus state: current_term: 1 leader_uuid: "ca3cdb05969c450da05a6b6e51550731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3cdb05969c450da05a6b6e51550731" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39743 } } }
I20260619 14:47:02.762861 21206 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:02.763222 21215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:02.767134 21215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:02.769551 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:02.763920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38311 (local address 127.0.0.1:39743)
0619 14:47:02.764133 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:02.764139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:02.764776 (+ 637us) server_negotiation.cc:407] Connection header received
0619 14:47:02.764898 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:02.764902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:02.764954 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:02.765038 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:02.765759 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:02.766532 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:02.767699 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:02.768043 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:02.768155 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:02.769090 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:02.769095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:02.769098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:02.769113 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:02.769143 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:02.769159 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:02.769240 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:02.769330 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 14:47:02.769387 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:47:02.773885 21215 catalog_manager.cc:1347] Generated new cluster ID: bf1b4606fce34a83a80b038e4ef4de43
I20260619 14:47:02.773962 21215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:02.798653 21215 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:02.800134 21215 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:02.814615 21215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca3cdb05969c450da05a6b6e51550731: Generated new TSK 0
I20260619 14:47:02.815543 21215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:47:02.824268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 21131
2026-06-19T14:47:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1493 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:02 dist-test-slave-cr22 krb5kdc[21230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:02 dist-test-slave-cr22 krb5kdc[21230](info): set up 2 sockets
Jun 19 14:47:02 dist-test-slave-cr22 krb5kdc[21230](info): commencing operation
krb5kdc: starting...
W20260619 14:47:03.911783 392 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:03 dist-test-slave-cr22 krb5kdc[21230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:47:04.043972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46071
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:04.164398 21243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:04.164690 21243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:04.164733 21243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:04.168833 21243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:04.168910 21243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:04.168929 21243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:04.168947 21243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:04.168962 21243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:04.174538 21243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:04.175668 21243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:04.176756 21243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:04.185542 21248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:04.185765 21243 server_base.cc:1034] running on GCE node
W20260619 14:47:04.185542 21249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:04.185575 21251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:04.186398 21243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:04.187500 21243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:04.188699 21243 hybrid_clock.cc:648] HybridClock initialized: now 1781880424188684 us; error 38 us; skew 500 ppm
Jun 19 14:47:04 dist-test-slave-cr22 krb5kdc[21230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:04.197247 21243 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:47:04.198228 21243 webserver.cc:460] Webserver started at http://127.0.0.1:46737/ using document root <none> and password file <none>
I20260619 14:47:04.198865 21243 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:04.198963 21243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:04.199199 21243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:04.201323 21243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "e5110037ea4f41699aff2a9a7afa89ee"
format_stamp: "Formatted at 2026-06-19 14:47:04 on dist-test-slave-cr22"
server_key: "f10c4a2b09a3e149403080bd18fe9e0e"
server_key_iv: "b15de35ba4a85a1080e7d4a97d57447e"
server_key_version: "encryptionkey@0"
I20260619 14:47:04.201891 21243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "e5110037ea4f41699aff2a9a7afa89ee"
format_stamp: "Formatted at 2026-06-19 14:47:04 on dist-test-slave-cr22"
server_key: "f10c4a2b09a3e149403080bd18fe9e0e"
server_key_iv: "b15de35ba4a85a1080e7d4a97d57447e"
server_key_version: "encryptionkey@0"
I20260619 14:47:04.205729 21243 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:47:04.208285 21258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:04.209735 21243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:04.209928 21243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "e5110037ea4f41699aff2a9a7afa89ee"
format_stamp: "Formatted at 2026-06-19 14:47:04 on dist-test-slave-cr22"
server_key: "f10c4a2b09a3e149403080bd18fe9e0e"
server_key_iv: "b15de35ba4a85a1080e7d4a97d57447e"
server_key_version: "encryptionkey@0"
I20260619 14:47:04.210057 21243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:04.215036 21243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:04.218364 21243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:04.218573 21243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:04.227239 21243 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46071
I20260619 14:47:04.227257 21310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46071 every 8 connection(s)
I20260619 14:47:04.228356 21243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:04.231125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 21243
I20260619 14:47:04.231357 21311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:04.231388 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:04.231703 392 external_mini_cluster.cc:1428] Setting key db2660012389cb636a1aaa9732d4b424
I20260619 14:47:04.237304 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.233125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38287 (local address 127.0.0.1:46071)
0619 14:47:04.233568 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:04.233586 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:04.233622 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:47:04.234501 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:04.234523 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:04.234832 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:04.235112 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:04.236081 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:04.236086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:04.236097 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:04.236108 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:04.236163 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:04.236182 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:04.236296 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:04.236401 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 14:47:04.236563 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":140,"threads_started":1}
I20260619 14:47:04.238397 21311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee: Bootstrap starting.
I20260619 14:47:04.240687 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:47:04.240859 21311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:04.241576 21311 log.cc:824] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:04.243958 21311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee: No bootstrap required, opened a new log
I20260619 14:47:04.244622 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.242996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34079 (local address 127.0.0.1:46071)
0619 14:47:04.243131 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:04.243136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:04.243334 (+ 198us) server_negotiation.cc:407] Connection header received
0619 14:47:04.243408 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:04.243411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:04.243467 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:04.243563 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:04.244236 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:04.244239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:04.244242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:04.244255 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:04.244284 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:04.244298 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:04.244409 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:04.244451 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 14:47:04.244490 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:47:04.247890 21311 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5110037ea4f41699aff2a9a7afa89ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46071 } }
I20260619 14:47:04.248133 21311 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:04.248171 21311 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5110037ea4f41699aff2a9a7afa89ee, State: Initialized, Role: FOLLOWER
I20260619 14:47:04.248554 21311 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5110037ea4f41699aff2a9a7afa89ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46071 } }
I20260619 14:47:04.248672 21311 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:04.248713 21311 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:04.248788 21311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:04.250046 21311 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5110037ea4f41699aff2a9a7afa89ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46071 } }
I20260619 14:47:04.250490 21311 leader_election.cc:302] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5110037ea4f41699aff2a9a7afa89ee; no voters:
I20260619 14:47:04.250924 21311 leader_election.cc:288] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:04.251101 21321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:04.251588 21321 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee [term 1 LEADER]: Becoming Leader. State: Replica: e5110037ea4f41699aff2a9a7afa89ee, State: Running, Role: LEADER
I20260619 14:47:04.252084 21321 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5110037ea4f41699aff2a9a7afa89ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46071 } }
I20260619 14:47:04.252612 21311 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:04.253705 21321 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5110037ea4f41699aff2a9a7afa89ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5110037ea4f41699aff2a9a7afa89ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46071 } } }
I20260619 14:47:04.253883 21321 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:04.254136 21321 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5110037ea4f41699aff2a9a7afa89ee. Latest consensus state: current_term: 1 leader_uuid: "e5110037ea4f41699aff2a9a7afa89ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5110037ea4f41699aff2a9a7afa89ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46071 } } }
I20260619 14:47:04.254297 21321 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee [sys.catalog]: This master's current role is: LEADER
W20260619 14:47:04.258523 21336 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:47:04.258668 21336 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:47:04.258848 21330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:04.259845 21330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:04.266023 21330 catalog_manager.cc:1347] Generated new cluster ID: eb9532409d7e46d8b231c6f5da231574
I20260619 14:47:04.266110 21330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:04.304557 21330 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:04.304944 21330 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:04.314234 21330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5110037ea4f41699aff2a9a7afa89ee: Generated new TSK 0
I20260619 14:47:04.315140 21330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:47:04.329175 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 21243
2026-06-19T14:47:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1504 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:04 dist-test-slave-cr22 krb5kdc[21342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:04 dist-test-slave-cr22 krb5kdc[21342](info): set up 2 sockets
Jun 19 14:47:04 dist-test-slave-cr22 krb5kdc[21342](info): commencing operation
krb5kdc: starting...
W20260619 14:47:05.392179 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:05 dist-test-slave-cr22 krb5kdc[21342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:47:05.519127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34447
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:05.631592 21355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:05.631836 21355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:05.631878 21355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:05.635871 21355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:05.635931 21355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:05.635949 21355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:05.635984 21355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:05.636000 21355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:05.642043 21355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:05.643256 21355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:05.644268 21355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:05.653275 21360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:05.653317 21361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:05.653378 21363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:05.653512 21355 server_base.cc:1034] running on GCE node
I20260619 14:47:05.654379 21355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:05.655725 21355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:05.656955 21355 hybrid_clock.cc:648] HybridClock initialized: now 1781880425656927 us; error 38 us; skew 500 ppm
Jun 19 14:47:05 dist-test-slave-cr22 krb5kdc[21342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:05.660024 21355 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:47:05.661016 21355 webserver.cc:460] Webserver started at http://127.0.0.1:45723/ using document root <none> and password file <none>
I20260619 14:47:05.661689 21355 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:05.661785 21355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:05.662016 21355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:05.664181 21355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "3bd4a83a7a3d4460b401cd35c5cda19a"
format_stamp: "Formatted at 2026-06-19 14:47:05 on dist-test-slave-cr22"
server_key: "d5f0bc53e1021958905a9585676fd348"
server_key_iv: "811ebb04e280bfe1fbffb49947b83d5b"
server_key_version: "encryptionkey@0"
I20260619 14:47:05.664878 21355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "3bd4a83a7a3d4460b401cd35c5cda19a"
format_stamp: "Formatted at 2026-06-19 14:47:05 on dist-test-slave-cr22"
server_key: "d5f0bc53e1021958905a9585676fd348"
server_key_iv: "811ebb04e280bfe1fbffb49947b83d5b"
server_key_version: "encryptionkey@0"
I20260619 14:47:05.668833 21355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:47:05.671653 21370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:05.673000 21355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:05.673161 21355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "3bd4a83a7a3d4460b401cd35c5cda19a"
format_stamp: "Formatted at 2026-06-19 14:47:05 on dist-test-slave-cr22"
server_key: "d5f0bc53e1021958905a9585676fd348"
server_key_iv: "811ebb04e280bfe1fbffb49947b83d5b"
server_key_version: "encryptionkey@0"
I20260619 14:47:05.673308 21355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:05.690567 21355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:05.693929 21355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:05.694150 21355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:05.702019 21355 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34447
I20260619 14:47:05.702045 21422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34447 every 8 connection(s)
I20260619 14:47:05.703038 21355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:05.705771 21423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:05.706110 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 21355
I20260619 14:47:05.706303 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:05.706585 392 external_mini_cluster.cc:1428] Setting key ffda9679cb283372ba70bfaf4d45f962
I20260619 14:47:05.711869 21423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a: Bootstrap starting.
I20260619 14:47:05.714443 21423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:05.715353 21423 log.cc:824] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:47:05 dist-test-slave-cr22 krb5kdc[21342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:47:05.717547 21423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a: No bootstrap required, opened a new log
I20260619 14:47:05.721359 21423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd4a83a7a3d4460b401cd35c5cda19a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34447 } }
I20260619 14:47:05.721606 21423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:05.721686 21423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd4a83a7a3d4460b401cd35c5cda19a, State: Initialized, Role: FOLLOWER
I20260619 14:47:05.721709 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:05.708027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:48809 (local address 127.0.0.1:34447)
0619 14:47:05.708472 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:05.708486 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:05.708520 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:47:05.709144 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:05.709159 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:05.709484 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:05.709762 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:05.710642 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.711719 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.712852 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.713353 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.713476 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:05.716460 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:05.716465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:05.716477 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:05.716504 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:05.718376 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:05.718889 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:05.718894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:05.718898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:05.719001 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:05.719340 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:05.719344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:05.719345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:05.719843 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:05.720127 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:05.720668 (+ 541us) server_negotiation.cc:299] Negotiation successful
0619 14:47:05.720883 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":145,"threads_started":1}
I20260619 14:47:05.722226 21423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd4a83a7a3d4460b401cd35c5cda19a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34447 } }
I20260619 14:47:05.722386 21423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:05.722450 21423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:05.722548 21423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:05.723536 21423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd4a83a7a3d4460b401cd35c5cda19a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34447 } }
I20260619 14:47:05.723873 21423 leader_election.cc:302] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bd4a83a7a3d4460b401cd35c5cda19a; no voters:
I20260619 14:47:05.724211 21423 leader_election.cc:288] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:05.724324 21428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:05.724522 21428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a [term 1 LEADER]: Becoming Leader. State: Replica: 3bd4a83a7a3d4460b401cd35c5cda19a, State: Running, Role: LEADER
I20260619 14:47:05.724812 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:47:05.725422 21423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:05.726739 21428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd4a83a7a3d4460b401cd35c5cda19a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34447 } }
I20260619 14:47:05.728621 21430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bd4a83a7a3d4460b401cd35c5cda19a. Latest consensus state: current_term: 1 leader_uuid: "3bd4a83a7a3d4460b401cd35c5cda19a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd4a83a7a3d4460b401cd35c5cda19a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34447 } } }
I20260619 14:47:05.728751 21430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:05.728598 21429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bd4a83a7a3d4460b401cd35c5cda19a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd4a83a7a3d4460b401cd35c5cda19a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34447 } } }
I20260619 14:47:05.729005 21429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a [sys.catalog]: This master's current role is: LEADER
W20260619 14:47:05.731199 21447 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:47:05.731299 21447 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:47:05.731408 21448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:05.732344 21448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:05.737366 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:05.726882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:60497 (local address 127.0.0.1:34447)
0619 14:47:05.727002 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:05.727006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:05.727853 (+ 847us) server_negotiation.cc:407] Connection header received
0619 14:47:05.728770 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:05.728773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:05.728824 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:05.728912 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:05.729837 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.730603 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.731763 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.732090 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.732198 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:05.733895 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:05.733899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:05.733902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:05.733926 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:05.735502 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:05.736000 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:05.736004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:05.736005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:05.736114 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:05.736475 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:05.736478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:05.736480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:05.736675 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:05.736840 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:05.737117 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 14:47:05.737214 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:47:05.738155 21448 catalog_manager.cc:1347] Generated new cluster ID: 6c762dd2900c496e81569afc00aa4dc1
I20260619 14:47:05.738209 21448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:05.744266 21448 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:05.745391 21448 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:05.754793 21448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bd4a83a7a3d4460b401cd35c5cda19a: Generated new TSK 0
I20260619 14:47:05.755795 21448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:47:05.768345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 21355
W20260619 14:47:05.774137 21340 connection.cc:535] client connection to 127.0.0.1:34447 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34447): Connection reset by peer (error 104)
2026-06-19T14:47:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:05 dist-test-slave-cr22 krb5kdc[21454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:05 dist-test-slave-cr22 krb5kdc[21454](info): set up 2 sockets
Jun 19 14:47:05 dist-test-slave-cr22 krb5kdc[21454](info): commencing operation
krb5kdc: starting...
W20260619 14:47:06.867877 392 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:06 dist-test-slave-cr22 krb5kdc[21454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:47:06.998973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40491
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:07.109524 21467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:07.109896 21467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:07.109987 21467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:07.114316 21467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:07.114439 21467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:07.114477 21467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:07.114517 21467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:07.114553 21467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:07.120131 21467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:07.121433 21467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:07.122470 21467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:07.132411 21473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:07.132370 21467 server_base.cc:1034] running on GCE node
W20260619 14:47:07.132355 21472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:07.132409 21475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:07.133301 21467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:07.134544 21467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:07.135752 21467 hybrid_clock.cc:648] HybridClock initialized: now 1781880427135726 us; error 44 us; skew 500 ppm
Jun 19 14:47:07 dist-test-slave-cr22 krb5kdc[21454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:07.138926 21467 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:47:07.139910 21467 webserver.cc:460] Webserver started at http://127.0.0.1:34963/ using document root <none> and password file <none>
I20260619 14:47:07.140483 21467 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:07.140547 21467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:07.140722 21467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:07.142903 21467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "b6ce4e38affb4519a1e3fe2db4f6ad32"
format_stamp: "Formatted at 2026-06-19 14:47:07 on dist-test-slave-cr22"
server_key: "eec57ef7313fac7c367ceeff8732e408"
server_key_iv: "8ba656b07895763637c06aa0e8f65a41"
server_key_version: "encryptionkey@0"
I20260619 14:47:07.143543 21467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "b6ce4e38affb4519a1e3fe2db4f6ad32"
format_stamp: "Formatted at 2026-06-19 14:47:07 on dist-test-slave-cr22"
server_key: "eec57ef7313fac7c367ceeff8732e408"
server_key_iv: "8ba656b07895763637c06aa0e8f65a41"
server_key_version: "encryptionkey@0"
I20260619 14:47:07.147732 21467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:47:07.150316 21482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:07.151845 21467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:07.151959 21467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "b6ce4e38affb4519a1e3fe2db4f6ad32"
format_stamp: "Formatted at 2026-06-19 14:47:07 on dist-test-slave-cr22"
server_key: "eec57ef7313fac7c367ceeff8732e408"
server_key_iv: "8ba656b07895763637c06aa0e8f65a41"
server_key_version: "encryptionkey@0"
I20260619 14:47:07.152068 21467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:07.177989 21467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:07.181437 21467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:07.181592 21467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:07.189560 21467 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40491
I20260619 14:47:07.189572 21534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40491 every 8 connection(s)
I20260619 14:47:07.190567 21467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:07.193450 21535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:07.196118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 21467
I20260619 14:47:07.196259 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:07.196540 392 external_mini_cluster.cc:1428] Setting key c4ef54dd1b1586561c56c4d5ad18ce22
I20260619 14:47:07.199714 21535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32: Bootstrap starting.
I20260619 14:47:07.201894 21535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:07.202616 21535 log.cc:824] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:07.204555 21535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32: No bootstrap required, opened a new log
I20260619 14:47:07.205569 21538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:07.197956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:33803 (local address 127.0.0.1:40491)
0619 14:47:07.198423 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:07.198438 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:07.198473 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:47:07.199156 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:07.199178 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:07.199631 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:07.199970 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:07.200384 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:07.201359 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:07.202314 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:07.202826 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:07.203002 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:07.203925 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:07.203930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:07.203942 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:07.204323 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:07.204375 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:07.204398 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:07.204548 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:07.204744 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 14:47:07.204885 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":121,"threads_started":1}
I20260619 14:47:07.208428 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:47:07.208425 21535 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ce4e38affb4519a1e3fe2db4f6ad32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40491 } }
I20260619 14:47:07.208703 21535 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:07.208768 21535 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ce4e38affb4519a1e3fe2db4f6ad32, State: Initialized, Role: FOLLOWER
I20260619 14:47:07.209200 21535 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ce4e38affb4519a1e3fe2db4f6ad32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40491 } }
I20260619 14:47:07.209349 21535 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:07.209414 21535 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:07.209514 21535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:07.210649 21535 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ce4e38affb4519a1e3fe2db4f6ad32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40491 } }
I20260619 14:47:07.211015 21535 leader_election.cc:302] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6ce4e38affb4519a1e3fe2db4f6ad32; no voters:
I20260619 14:47:07.211349 21535 leader_election.cc:288] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:07.211469 21545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:07.211668 21545 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32 [term 1 LEADER]: Becoming Leader. State: Replica: b6ce4e38affb4519a1e3fe2db4f6ad32, State: Running, Role: LEADER
I20260619 14:47:07.212008 21545 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ce4e38affb4519a1e3fe2db4f6ad32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40491 } }
I20260619 14:47:07.212673 21535 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:07.213644 21547 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6ce4e38affb4519a1e3fe2db4f6ad32. Latest consensus state: current_term: 1 leader_uuid: "b6ce4e38affb4519a1e3fe2db4f6ad32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ce4e38affb4519a1e3fe2db4f6ad32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40491 } } }
I20260619 14:47:07.213778 21547 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:07.214080 21546 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6ce4e38affb4519a1e3fe2db4f6ad32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ce4e38affb4519a1e3fe2db4f6ad32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40491 } } }
I20260619 14:47:07.214231 21546 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:07.215804 21538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:07.210774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:43627 (local address 127.0.0.1:40491)
0619 14:47:07.210888 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:07.210892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:07.211136 (+ 244us) server_negotiation.cc:407] Connection header received
0619 14:47:07.211179 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:07.211182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:07.211233 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:07.211318 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:07.211637 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:07.212440 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:07.213566 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:07.213917 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:07.214019 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:07.215152 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:07.215155 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:07.215158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:07.215172 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:07.215203 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:07.215219 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:07.215309 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:07.215576 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 14:47:07.215615 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 14:47:07.217711 21561 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:47:07.217773 21561 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:47:07.217855 21554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:07.218520 21554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:07.224040 21554 catalog_manager.cc:1347] Generated new cluster ID: 5c36c31d475a4b95a797089cfc65ccf6
I20260619 14:47:07.224126 21554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:07.238219 21554 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:07.239234 21554 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:07.244112 21554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6ce4e38affb4519a1e3fe2db4f6ad32: Generated new TSK 0
I20260619 14:47:07.244823 21554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:47:07.252995 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 21467
W20260619 14:47:07.258535 21452 connection.cc:535] client connection to 127.0.0.1:40491 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40491): Connection reset by peer (error 104)
2026-06-19T14:47:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1484 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:07 dist-test-slave-cr22 krb5kdc[21566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:07 dist-test-slave-cr22 krb5kdc[21566](info): set up 2 sockets
Jun 19 14:47:07 dist-test-slave-cr22 krb5kdc[21566](info): commencing operation
krb5kdc: starting...
W20260619 14:47:08.332600 392 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:08 dist-test-slave-cr22 krb5kdc[21566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:47:08.472692 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:08.607908 21579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:08.608311 21579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:08.608359 21579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:08.612638 21579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:08.612716 21579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:08.612736 21579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:08.612754 21579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:08.612771 21579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:08.618592 21579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:08.619788 21579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:08.621028 21579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:08.631271 21585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:08.631397 21587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:08.631479 21579 server_base.cc:1034] running on GCE node
W20260619 14:47:08.631271 21584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:08.632225 21579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:08.633351 21579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:08.634572 21579 hybrid_clock.cc:648] HybridClock initialized: now 1781880428634538 us; error 47 us; skew 500 ppm
Jun 19 14:47:08 dist-test-slave-cr22 krb5kdc[21566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:08.637800 21579 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:47:08.638727 21579 webserver.cc:460] Webserver started at http://127.0.0.1:35857/ using document root <none> and password file <none>
I20260619 14:47:08.639357 21579 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:08.639423 21579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:08.639701 21579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:08.641877 21579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "0dd0fd77ee7c4f0e86d044502d6ba524"
format_stamp: "Formatted at 2026-06-19 14:47:08 on dist-test-slave-cr22"
server_key: "f5e258c77b90a368805940b1b8692a19"
server_key_iv: "330bb29b920ac36e86e72bdd693d1049"
server_key_version: "encryptionkey@0"
I20260619 14:47:08.642480 21579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "0dd0fd77ee7c4f0e86d044502d6ba524"
format_stamp: "Formatted at 2026-06-19 14:47:08 on dist-test-slave-cr22"
server_key: "f5e258c77b90a368805940b1b8692a19"
server_key_iv: "330bb29b920ac36e86e72bdd693d1049"
server_key_version: "encryptionkey@0"
I20260619 14:47:08.646644 21579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 14:47:08.649609 21594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:08.651214 21579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:47:08.651374 21579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "0dd0fd77ee7c4f0e86d044502d6ba524"
format_stamp: "Formatted at 2026-06-19 14:47:08 on dist-test-slave-cr22"
server_key: "f5e258c77b90a368805940b1b8692a19"
server_key_iv: "330bb29b920ac36e86e72bdd693d1049"
server_key_version: "encryptionkey@0"
I20260619 14:47:08.651559 21579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:08.656404 21579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:08.660074 21579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:08.660281 21579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:08.668598 21579 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45369
I20260619 14:47:08.668619 21646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45369 every 8 connection(s)
I20260619 14:47:08.669538 21579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:08.671128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 21579
I20260619 14:47:08.671294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:08.671646 392 external_mini_cluster.cc:1428] Setting key dfc872ed51ba8942aa736a9b92430033
I20260619 14:47:08.672477 21647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:08.676882 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:08.673039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:40775 (local address 127.0.0.1:45369)
0619 14:47:08.673470 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:08.673485 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:08.673529 (+ 44us) server_negotiation.cc:407] Connection header received
0619 14:47:08.674225 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:08.674241 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:08.674516 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:08.674842 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:08.675671 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:08.675676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:08.675686 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:08.675696 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:08.675744 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:08.675762 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:08.675906 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:08.676082 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 14:47:08.676230 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":134,"threads_started":1}
I20260619 14:47:08.678771 21647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524: Bootstrap starting.
I20260619 14:47:08.679764 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:47:08.681259 21647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:08.682117 21647 log.cc:824] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:08.683759 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:08.681807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:49649 (local address 127.0.0.1:45369)
0619 14:47:08.681934 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:08.681938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:08.682179 (+ 241us) server_negotiation.cc:407] Connection header received
0619 14:47:08.682287 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:08.682290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:08.682329 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:08.682420 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:08.683187 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:08.683197 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:08.683200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:08.683212 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:08.683240 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:08.683253 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:08.683347 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:08.683583 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 14:47:08.683624 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:47:08.684485 21647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524: No bootstrap required, opened a new log
I20260619 14:47:08.688011 21647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd0fd77ee7c4f0e86d044502d6ba524" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } }
I20260619 14:47:08.688236 21647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:08.688272 21647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dd0fd77ee7c4f0e86d044502d6ba524, State: Initialized, Role: FOLLOWER
I20260619 14:47:08.688639 21647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd0fd77ee7c4f0e86d044502d6ba524" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } }
I20260619 14:47:08.688757 21647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:08.688797 21647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:08.688868 21647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:08.689838 21647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd0fd77ee7c4f0e86d044502d6ba524" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } }
I20260619 14:47:08.690136 21647 leader_election.cc:302] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dd0fd77ee7c4f0e86d044502d6ba524; no voters:
I20260619 14:47:08.690404 21647 leader_election.cc:288] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:08.690538 21657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:08.690805 21657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524 [term 1 LEADER]: Becoming Leader. State: Replica: 0dd0fd77ee7c4f0e86d044502d6ba524, State: Running, Role: LEADER
I20260619 14:47:08.691145 21657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd0fd77ee7c4f0e86d044502d6ba524" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } }
I20260619 14:47:08.691746 21647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:08.692903 21659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dd0fd77ee7c4f0e86d044502d6ba524. Latest consensus state: current_term: 1 leader_uuid: "0dd0fd77ee7c4f0e86d044502d6ba524" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd0fd77ee7c4f0e86d044502d6ba524" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } } }
I20260619 14:47:08.693041 21659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:08.693456 21666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:08.696791 21666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:08.693440 21658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dd0fd77ee7c4f0e86d044502d6ba524" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd0fd77ee7c4f0e86d044502d6ba524" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } } }
I20260619 14:47:08.699146 21658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:08.702574 21666 catalog_manager.cc:1347] Generated new cluster ID: 7768ee85588b41788b313d7b1a0a9453
I20260619 14:47:08.702682 21666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:08.713989 21666 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:08.714363 21666 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:08.726979 21666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dd0fd77ee7c4f0e86d044502d6ba524: Generated new TSK 0
I20260619 14:47:08.727947 21666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:47:08.733278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 21579
2026-06-19T14:47:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1480 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:08 dist-test-slave-cr22 krb5kdc[21678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:08 dist-test-slave-cr22 krb5kdc[21678](info): set up 2 sockets
Jun 19 14:47:08 dist-test-slave-cr22 krb5kdc[21678](info): commencing operation
krb5kdc: starting...
W20260619 14:47:09.807897 392 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:09 dist-test-slave-cr22 krb5kdc[21678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:47:09.938905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:10.056679 21691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:10.057034 21691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:10.057122 21691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:10.061482 21691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:10.061587 21691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:10.061625 21691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:10.061683 21691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:10.061719 21691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:10.067406 21691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35729
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:10.068617 21691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:10.069658 21691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:10.078542 21696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:10.078563 21699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:10.078589 21691 server_base.cc:1034] running on GCE node
W20260619 14:47:10.078542 21697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:10.079418 21691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:10.080610 21691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:10.081820 21691 hybrid_clock.cc:648] HybridClock initialized: now 1781880430081790 us; error 43 us; skew 500 ppm
Jun 19 14:47:10 dist-test-slave-cr22 krb5kdc[21678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:10.090816 21691 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:47:10.091882 21691 webserver.cc:460] Webserver started at http://127.0.0.1:33527/ using document root <none> and password file <none>
I20260619 14:47:10.092473 21691 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:10.092568 21691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:10.092810 21691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:10.094926 21691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "c83306882dbc4d598e7996d3a4df8728"
format_stamp: "Formatted at 2026-06-19 14:47:10 on dist-test-slave-cr22"
server_key: "2b3933c48333c85a71844cc69e22d9a0"
server_key_iv: "7c9e25da7b87a4482f2791bb834a5899"
server_key_version: "encryptionkey@0"
I20260619 14:47:10.095613 21691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "c83306882dbc4d598e7996d3a4df8728"
format_stamp: "Formatted at 2026-06-19 14:47:10 on dist-test-slave-cr22"
server_key: "2b3933c48333c85a71844cc69e22d9a0"
server_key_iv: "7c9e25da7b87a4482f2791bb834a5899"
server_key_version: "encryptionkey@0"
I20260619 14:47:10.099774 21691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:47:10.102396 21706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:10.103880 21691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:10.104043 21691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "c83306882dbc4d598e7996d3a4df8728"
format_stamp: "Formatted at 2026-06-19 14:47:10 on dist-test-slave-cr22"
server_key: "2b3933c48333c85a71844cc69e22d9a0"
server_key_iv: "7c9e25da7b87a4482f2791bb834a5899"
server_key_version: "encryptionkey@0"
I20260619 14:47:10.104177 21691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:10.124648 21691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:10.128329 21691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:10.128659 21691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:10.136729 21691 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35729
I20260619 14:47:10.136945 21758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35729 every 8 connection(s)
I20260619 14:47:10.137975 21691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:10.141546 21759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:10.146957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 21691
I20260619 14:47:10.147120 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:10.147423 392 external_mini_cluster.cc:1428] Setting key 011319eea919e2705bae66ecb408f38a
I20260619 14:47:10.148221 21759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728: Bootstrap starting.
I20260619 14:47:10.150815 21759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:10.151711 21759 log.cc:824] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:10.153736 21759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728: No bootstrap required, opened a new log
Jun 19 14:47:10 dist-test-slave-cr22 krb5kdc[21678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:47:10.158921 21759 raft_consensus.cc:348] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83306882dbc4d598e7996d3a4df8728" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35729 } }
I20260619 14:47:10.159212 21759 raft_consensus.cc:374] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:10.159310 21759 raft_consensus.cc:729] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c83306882dbc4d598e7996d3a4df8728, State: Initialized, Role: FOLLOWER
I20260619 14:47:10.159860 21759 consensus_queue.cc:260] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83306882dbc4d598e7996d3a4df8728" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35729 } }
I20260619 14:47:10.160032 21759 raft_consensus.cc:388] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:10.160122 21759 raft_consensus.cc:482] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:10.160260 21759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:10.161289 21759 raft_consensus.cc:504] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83306882dbc4d598e7996d3a4df8728" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35729 } }
I20260619 14:47:10.161660 21759 leader_election.cc:302] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c83306882dbc4d598e7996d3a4df8728; no voters:
I20260619 14:47:10.162026 21759 leader_election.cc:288] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:10.162174 21764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:10.162458 21764 raft_consensus.cc:686] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728 [term 1 LEADER]: Becoming Leader. State: Replica: c83306882dbc4d598e7996d3a4df8728, State: Running, Role: LEADER
I20260619 14:47:10.162822 21764 consensus_queue.cc:237] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83306882dbc4d598e7996d3a4df8728" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35729 } }
I20260619 14:47:10.163594 21759 sys_catalog.cc:564] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:10.164558 21765 sys_catalog.cc:455] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c83306882dbc4d598e7996d3a4df8728. Latest consensus state: current_term: 1 leader_uuid: "c83306882dbc4d598e7996d3a4df8728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83306882dbc4d598e7996d3a4df8728" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35729 } } }
I20260619 14:47:10.164685 21765 sys_catalog.cc:458] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:10.165055 21766 sys_catalog.cc:455] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c83306882dbc4d598e7996d3a4df8728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83306882dbc4d598e7996d3a4df8728" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35729 } } }
I20260619 14:47:10.165191 21766 sys_catalog.cc:458] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:10.165509 21773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:10.165560 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:10.148946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:49471 (local address 127.0.0.1:35729)
0619 14:47:10.149424 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:10.149439 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:10.149476 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:47:10.150204 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:10.150225 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:10.150575 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:10.150933 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:10.152299 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.153795 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.155186 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.155855 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.156025 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:10.159284 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:10.159291 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:10.159393 (+ 102us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:10.159466 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:10.161643 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:10.162259 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:10.162266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:10.162272 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:10.162438 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:10.162908 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:10.162913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:10.162916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:10.163490 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:10.163695 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:10.164217 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 14:47:10.164505 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":150,"threads_started":1}
I20260619 14:47:10.169250 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:47:10.169394 21773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:10.175982 21773 catalog_manager.cc:1347] Generated new cluster ID: b4362e270c7c4d7f899d095e64a9d083
I20260619 14:47:10.176046 21773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:10.182700 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:10.171629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:35873 (local address 127.0.0.1:35729)
0619 14:47:10.171745 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:10.171750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:10.174268 (+ 2518us) server_negotiation.cc:407] Connection header received
0619 14:47:10.174467 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:10.174471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:10.174525 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:10.174637 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:10.175320 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.176113 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.177201 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.177561 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.177678 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:10.179262 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:10.179267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:10.179269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:10.179295 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:10.180934 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:10.181446 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:10.181449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:10.181452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:10.181538 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:10.181860 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:10.181863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:10.181865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:10.182064 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:10.182216 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:10.182446 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 14:47:10.182544 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:47:10.186517 21773 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:10.187747 21773 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:10.200003 21773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c83306882dbc4d598e7996d3a4df8728: Generated new TSK 0
I20260619 14:47:10.200775 21773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:47:10.209713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 21691
W20260619 14:47:10.214985 21676 connection.cc:535] client connection to 127.0.0.1:35729 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35729): Connection reset by peer (error 104)
2026-06-19T14:47:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1475 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:10 dist-test-slave-cr22 krb5kdc[21790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:10 dist-test-slave-cr22 krb5kdc[21790](info): set up 2 sockets
Jun 19 14:47:10 dist-test-slave-cr22 krb5kdc[21790](info): commencing operation
krb5kdc: starting...
W20260619 14:47:11.283946 392 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:11 dist-test-slave-cr22 krb5kdc[21790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:47:11.414330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:11.528057 21803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:11.528347 21803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:11.528391 21803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:11.532744 21803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:11.532837 21803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:11.532876 21803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:11.532899 21803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:11.532996 21803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:11.538758 21803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41777
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:11.539927 21803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:11.540993 21803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:11.549961 21808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:11.549979 21809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:11.549975 21811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:11.550035 21803 server_base.cc:1034] running on GCE node
I20260619 14:47:11.551088 21803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:11.552326 21803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:11.553565 21803 hybrid_clock.cc:648] HybridClock initialized: now 1781880431553529 us; error 48 us; skew 500 ppm
Jun 19 14:47:11 dist-test-slave-cr22 krb5kdc[21790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:11.556734 21803 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:47:11.557765 21803 webserver.cc:460] Webserver started at http://127.0.0.1:46651/ using document root <none> and password file <none>
I20260619 14:47:11.558362 21803 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:11.558454 21803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:11.558696 21803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:11.561143 21803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "515d26c08f954433975bf68eb878aa0d"
format_stamp: "Formatted at 2026-06-19 14:47:11 on dist-test-slave-cr22"
server_key: "aadef6127095b958b212a7208195e215"
server_key_iv: "403964add432a17531b6d4a2500dadc9"
server_key_version: "encryptionkey@0"
I20260619 14:47:11.561851 21803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "515d26c08f954433975bf68eb878aa0d"
format_stamp: "Formatted at 2026-06-19 14:47:11 on dist-test-slave-cr22"
server_key: "aadef6127095b958b212a7208195e215"
server_key_iv: "403964add432a17531b6d4a2500dadc9"
server_key_version: "encryptionkey@0"
I20260619 14:47:11.565928 21803 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:47:11.568904 21818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:11.570286 21803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:47:11.570441 21803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "515d26c08f954433975bf68eb878aa0d"
format_stamp: "Formatted at 2026-06-19 14:47:11 on dist-test-slave-cr22"
server_key: "aadef6127095b958b212a7208195e215"
server_key_iv: "403964add432a17531b6d4a2500dadc9"
server_key_version: "encryptionkey@0"
I20260619 14:47:11.570573 21803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:11.598618 21803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:11.602151 21803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:11.602365 21803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:11.610001 21803 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41777
I20260619 14:47:11.610080 21870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41777 every 8 connection(s)
I20260619 14:47:11.611050 21803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:11.611725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 21803
I20260619 14:47:11.611896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:11.612178 392 external_mini_cluster.cc:1428] Setting key 80f4dc385abf937298388d0aabbfc83f
I20260619 14:47:11.614929 21871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:11.621492 21871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d: Bootstrap starting.
I20260619 14:47:11.622102 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:11.613627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:34339 (local address 127.0.0.1:41777)
0619 14:47:11.614209 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:11.614224 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:11.614263 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:47:11.615316 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:11.615336 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:11.615774 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:11.616155 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:11.616559 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:11.617550 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:11.618478 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:11.618959 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:11.619144 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:11.620251 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:11.620256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:11.620269 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:11.620682 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:11.620738 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:11.620769 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:11.620940 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:11.621105 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 14:47:11.621256 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":265,"threads_started":1}
I20260619 14:47:11.624346 21871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:11.625385 21871 log.cc:824] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:11.625563 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:47:11.628026 21871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d: No bootstrap required, opened a new log
I20260619 14:47:11.632634 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:11.627974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:49701 (local address 127.0.0.1:41777)
0619 14:47:11.628126 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:11.628131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:11.628275 (+ 144us) server_negotiation.cc:407] Connection header received
0619 14:47:11.628356 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:11.628359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:11.628409 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:11.628567 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:11.628867 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:11.629632 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:11.630575 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:11.630914 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:11.631026 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:11.632150 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:11.632155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:11.632158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:11.632174 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:11.632203 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:11.632216 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:11.632312 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:11.632439 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 14:47:11.632479 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:47:11.633186 21871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515d26c08f954433975bf68eb878aa0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41777 } }
I20260619 14:47:11.633493 21871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:11.633584 21871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515d26c08f954433975bf68eb878aa0d, State: Initialized, Role: FOLLOWER
I20260619 14:47:11.634117 21871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515d26c08f954433975bf68eb878aa0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41777 } }
I20260619 14:47:11.634268 21871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:11.634356 21871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:11.634449 21871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:11.635535 21871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515d26c08f954433975bf68eb878aa0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41777 } }
I20260619 14:47:11.635896 21871 leader_election.cc:302] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 515d26c08f954433975bf68eb878aa0d; no voters:
I20260619 14:47:11.636256 21871 leader_election.cc:288] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:11.636344 21881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:11.636546 21881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d [term 1 LEADER]: Becoming Leader. State: Replica: 515d26c08f954433975bf68eb878aa0d, State: Running, Role: LEADER
I20260619 14:47:11.636868 21881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515d26c08f954433975bf68eb878aa0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41777 } }
I20260619 14:47:11.637661 21871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:11.638731 21882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "515d26c08f954433975bf68eb878aa0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515d26c08f954433975bf68eb878aa0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41777 } } }
I20260619 14:47:11.638860 21882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:11.639233 21883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 515d26c08f954433975bf68eb878aa0d. Latest consensus state: current_term: 1 leader_uuid: "515d26c08f954433975bf68eb878aa0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515d26c08f954433975bf68eb878aa0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41777 } } }
I20260619 14:47:11.639359 21883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:11.639272 21890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:11.642876 21890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:11.648705 21890 catalog_manager.cc:1347] Generated new cluster ID: e3ac11428e2e4ccbb9c993aef2410d1d
I20260619 14:47:11.648801 21890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:11.680217 21890 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:11.681234 21890 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:11.694591 21890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 515d26c08f954433975bf68eb878aa0d: Generated new TSK 0
I20260619 14:47:11.695369 21890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:47:11.700464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 21803
W20260619 14:47:11.705689 21788 connection.cc:535] client connection to 127.0.0.1:41777 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41777): Connection reset by peer (error 104)
2026-06-19T14:47:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1490 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:11 dist-test-slave-cr22 krb5kdc[21902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:11 dist-test-slave-cr22 krb5kdc[21902](info): set up 2 sockets
Jun 19 14:47:11 dist-test-slave-cr22 krb5kdc[21902](info): commencing operation
krb5kdc: starting...
W20260619 14:47:12.787859 392 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:12 dist-test-slave-cr22 krb5kdc[21902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:47:12.915030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33071
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:13.025468 21915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:13.025790 21915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:13.025873 21915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:13.030112 21915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:13.030227 21915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:13.030264 21915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:13.030303 21915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:13.030342 21915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:13.035893 21915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33071
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:13.037202 21915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:13.038239 21915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:13.047226 21920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:13.047253 21923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:13.047226 21921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:13.047333 21915 server_base.cc:1034] running on GCE node
I20260619 14:47:13.048029 21915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:13.049396 21915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:13.050655 21915 hybrid_clock.cc:648] HybridClock initialized: now 1781880433050617 us; error 53 us; skew 500 ppm
Jun 19 14:47:13 dist-test-slave-cr22 krb5kdc[21902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:13.053849 21915 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:47:13.054939 21915 webserver.cc:460] Webserver started at http://127.0.0.1:38003/ using document root <none> and password file <none>
I20260619 14:47:13.055632 21915 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:13.055727 21915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:13.055958 21915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:13.058398 21915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "aab748d388fd4fcd921bac5d2b6f53a0"
format_stamp: "Formatted at 2026-06-19 14:47:13 on dist-test-slave-cr22"
server_key: "af9f249bf124f21b497522769e316e48"
server_key_iv: "fb1af744fff0c6d2c370ccdbc6810713"
server_key_version: "encryptionkey@0"
I20260619 14:47:13.059007 21915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "aab748d388fd4fcd921bac5d2b6f53a0"
format_stamp: "Formatted at 2026-06-19 14:47:13 on dist-test-slave-cr22"
server_key: "af9f249bf124f21b497522769e316e48"
server_key_iv: "fb1af744fff0c6d2c370ccdbc6810713"
server_key_version: "encryptionkey@0"
I20260619 14:47:13.062835 21915 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:47:13.065623 21930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:13.066989 21915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:47:13.067148 21915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "aab748d388fd4fcd921bac5d2b6f53a0"
format_stamp: "Formatted at 2026-06-19 14:47:13 on dist-test-slave-cr22"
server_key: "af9f249bf124f21b497522769e316e48"
server_key_iv: "fb1af744fff0c6d2c370ccdbc6810713"
server_key_version: "encryptionkey@0"
I20260619 14:47:13.067294 21915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:13.072582 21915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:13.075973 21915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:13.076184 21915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:13.083474 21915 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33071
I20260619 14:47:13.083480 21982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33071 every 8 connection(s)
I20260619 14:47:13.084430 21915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:13.087334 21983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:13.091833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 21915
I20260619 14:47:13.091981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:13.092245 392 external_mini_cluster.cc:1428] Setting key 85b50eb1db0ed831635f085cb41b4462
I20260619 14:47:13.093948 21983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0: Bootstrap starting.
I20260619 14:47:13.096652 21983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:13.097582 21983 log.cc:824] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:13.098212 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:13.093965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:44783 (local address 127.0.0.1:33071)
0619 14:47:13.094447 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:13.094462 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:13.094500 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:47:13.095181 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:13.095197 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:13.095586 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:13.095946 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:13.096893 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:13.096897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:13.096909 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:13.096921 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:13.096974 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:13.096995 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:13.097085 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:13.097231 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 14:47:13.097420 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":146,"threads_started":1}
I20260619 14:47:13.099862 21983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0: No bootstrap required, opened a new log
I20260619 14:47:13.102131 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:47:13.104766 21983 raft_consensus.cc:348] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab748d388fd4fcd921bac5d2b6f53a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33071 } }
I20260619 14:47:13.105062 21983 raft_consensus.cc:374] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:13.105147 21983 raft_consensus.cc:729] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab748d388fd4fcd921bac5d2b6f53a0, State: Initialized, Role: FOLLOWER
I20260619 14:47:13.105690 21983 consensus_queue.cc:260] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab748d388fd4fcd921bac5d2b6f53a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33071 } }
I20260619 14:47:13.105880 21983 raft_consensus.cc:388] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:13.105962 21983 raft_consensus.cc:482] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:13.106067 21983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:13.106832 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:13.104724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:52661 (local address 127.0.0.1:33071)
0619 14:47:13.104885 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:13.104891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:13.105382 (+ 491us) server_negotiation.cc:407] Connection header received
0619 14:47:13.105531 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:13.105534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:13.105574 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:13.105663 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:13.106459 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:13.106462 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:13.106465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:13.106477 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:13.106507 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:13.106520 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:13.106610 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:13.106674 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 14:47:13.106716 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:47:13.107429 21983 raft_consensus.cc:504] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab748d388fd4fcd921bac5d2b6f53a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33071 } }
I20260619 14:47:13.107936 21983 leader_election.cc:302] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aab748d388fd4fcd921bac5d2b6f53a0; no voters:
I20260619 14:47:13.108285 21983 leader_election.cc:288] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:13.108353 21993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:13.108572 21993 raft_consensus.cc:686] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0 [term 1 LEADER]: Becoming Leader. State: Replica: aab748d388fd4fcd921bac5d2b6f53a0, State: Running, Role: LEADER
I20260619 14:47:13.108922 21993 consensus_queue.cc:237] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab748d388fd4fcd921bac5d2b6f53a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33071 } }
I20260619 14:47:13.109973 21983 sys_catalog.cc:564] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:13.111308 21994 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aab748d388fd4fcd921bac5d2b6f53a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab748d388fd4fcd921bac5d2b6f53a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33071 } } }
I20260619 14:47:13.111476 21994 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:13.111686 21995 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aab748d388fd4fcd921bac5d2b6f53a0. Latest consensus state: current_term: 1 leader_uuid: "aab748d388fd4fcd921bac5d2b6f53a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab748d388fd4fcd921bac5d2b6f53a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33071 } } }
I20260619 14:47:13.111788 21995 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:13.111866 22002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:13.115125 22002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:13.120977 22002 catalog_manager.cc:1347] Generated new cluster ID: 56630af4ba584efdaebf0963740a5c9f
I20260619 14:47:13.121085 22002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:13.146165 22002 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:13.146528 22002 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:13.160916 22002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aab748d388fd4fcd921bac5d2b6f53a0: Generated new TSK 0
I20260619 14:47:13.161779 22002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:47:13.174499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 21915
2026-06-19T14:47:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1474 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:13 dist-test-slave-cr22 krb5kdc[22014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:13 dist-test-slave-cr22 krb5kdc[22014](info): set up 2 sockets
Jun 19 14:47:13 dist-test-slave-cr22 krb5kdc[22014](info): commencing operation
krb5kdc: starting...
W20260619 14:47:14.231891 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:14 dist-test-slave-cr22 krb5kdc[22014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:47:14.366216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42053
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:14.476465 22027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:14.476795 22027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:14.476871 22027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:14.480981 22027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:14.481089 22027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:14.481127 22027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:14.481194 22027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:14.481248 22027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:14.486660 22027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:14.487867 22027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:14.488906 22027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:14.498860 22033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:14.498853 22027 server_base.cc:1034] running on GCE node
W20260619 14:47:14.498873 22032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:14.498847 22035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:14.499845 22027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:14.501129 22027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:14.502317 22027 hybrid_clock.cc:648] HybridClock initialized: now 1781880434502289 us; error 45 us; skew 500 ppm
Jun 19 14:47:14 dist-test-slave-cr22 krb5kdc[22014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:14.505966 22027 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:47:14.507102 22027 webserver.cc:460] Webserver started at http://127.0.98.62:46143/ using document root <none> and password file <none>
I20260619 14:47:14.507823 22027 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:14.507898 22027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:14.508136 22027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:14.510428 22027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "190f40e39729450a9800ceee4c940f21"
format_stamp: "Formatted at 2026-06-19 14:47:14 on dist-test-slave-cr22"
server_key: "67f19ad9ba0df4d1b90dda03c1decc34"
server_key_iv: "d93e65bf7f7963348931c8bd6bd984ce"
server_key_version: "encryptionkey@0"
I20260619 14:47:14.511111 22027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "190f40e39729450a9800ceee4c940f21"
format_stamp: "Formatted at 2026-06-19 14:47:14 on dist-test-slave-cr22"
server_key: "67f19ad9ba0df4d1b90dda03c1decc34"
server_key_iv: "d93e65bf7f7963348931c8bd6bd984ce"
server_key_version: "encryptionkey@0"
I20260619 14:47:14.515534 22027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260619 14:47:14.518345 22042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:14.520198 22027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:47:14.520355 22027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "190f40e39729450a9800ceee4c940f21"
format_stamp: "Formatted at 2026-06-19 14:47:14 on dist-test-slave-cr22"
server_key: "67f19ad9ba0df4d1b90dda03c1decc34"
server_key_iv: "d93e65bf7f7963348931c8bd6bd984ce"
server_key_version: "encryptionkey@0"
I20260619 14:47:14.520486 22027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:14.545915 22027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:14.549383 22027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:14.549615 22027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:14.558019 22027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42053
I20260619 14:47:14.558074 22094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42053 every 8 connection(s)
I20260619 14:47:14.559152 22027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:14.562049 22095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:14.563176 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 22027
I20260619 14:47:14.563319 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:14.563650 392 external_mini_cluster.cc:1428] Setting key 4ddbb0f39027defb9327f029ebf4e61e
I20260619 14:47:14.568668 22095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21: Bootstrap starting.
I20260619 14:47:14.571260 22095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:14.572182 22095 log.cc:824] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:14.574522 22095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21: No bootstrap required, opened a new log
Jun 19 14:47:14 dist-test-slave-cr22 krb5kdc[22014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:47:14.578783 22095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190f40e39729450a9800ceee4c940f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42053 } }
I20260619 14:47:14.579080 22095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:14.579133 22095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190f40e39729450a9800ceee4c940f21, State: Initialized, Role: FOLLOWER
I20260619 14:47:14.579775 22095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190f40e39729450a9800ceee4c940f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42053 } }
I20260619 14:47:14.579955 22095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:14.580058 22095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:14.580163 22095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:14.581394 22095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190f40e39729450a9800ceee4c940f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42053 } }
I20260619 14:47:14.581748 22095 leader_election.cc:302] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 190f40e39729450a9800ceee4c940f21; no voters:
I20260619 14:47:14.582155 22095 leader_election.cc:288] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:14.582422 22100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:14.582679 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:14.565151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34557 (local address 127.0.98.62:42053)
0619 14:47:14.565604 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:14.565620 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:14.565657 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:47:14.566324 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:14.566341 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:14.566701 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:14.566988 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:14.567930 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:14.569252 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:14.570725 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:14.571759 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:14.571896 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:14.575535 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:14.575542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:14.575559 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:14.575590 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:14.578326 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:14.578968 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:14.578974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:14.578980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:14.579195 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:14.580223 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:14.580226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:14.580228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:14.580760 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:14.580991 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:14.581347 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 14:47:14.581592 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":145,"threads_started":1}
I20260619 14:47:14.582912 22100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21 [term 1 LEADER]: Becoming Leader. State: Replica: 190f40e39729450a9800ceee4c940f21, State: Running, Role: LEADER
I20260619 14:47:14.583340 22100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190f40e39729450a9800ceee4c940f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42053 } }
I20260619 14:47:14.583424 22095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:14.584800 22100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 190f40e39729450a9800ceee4c940f21. Latest consensus state: current_term: 1 leader_uuid: "190f40e39729450a9800ceee4c940f21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190f40e39729450a9800ceee4c940f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42053 } } }
I20260619 14:47:14.584934 22100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:14.585436 22112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:14.586057 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:47:14.585435 22101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "190f40e39729450a9800ceee4c940f21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190f40e39729450a9800ceee4c940f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42053 } } }
I20260619 14:47:14.586526 22101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:14.589159 22112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:14.595304 22112 catalog_manager.cc:1347] Generated new cluster ID: 78d6f493ecec4a2cbb8b7b8123ab1dbe
I20260619 14:47:14.595363 22112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:14.597394 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:14.588336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37991 (local address 127.0.98.62:42053)
0619 14:47:14.588447 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:14.588451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:14.588681 (+ 230us) server_negotiation.cc:407] Connection header received
0619 14:47:14.588819 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:14.588821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:14.588873 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:14.588958 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:14.589678 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:14.590196 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:14.590999 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:14.591320 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:14.591416 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:14.593375 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:14.593380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:14.593384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:14.593410 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:14.595060 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:14.595653 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:14.595656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:14.595658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:14.595742 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:14.596162 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:14.596165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:14.596167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:14.596360 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:14.597071 (+ 711us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:14.597158 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 14:47:14.597244 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:47:14.607779 22112 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:14.608873 22112 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:14.618912 22112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 190f40e39729450a9800ceee4c940f21: Generated new TSK 0
I20260619 14:47:14.619774 22112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:47:14.628957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 22027
W20260619 14:47:14.634281 22012 connection.cc:535] client connection to 127.0.98.62:42053 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:42053): Connection reset by peer (error 104)
2026-06-19T14:47:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:14 dist-test-slave-cr22 krb5kdc[22126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:14 dist-test-slave-cr22 krb5kdc[22126](info): set up 2 sockets
Jun 19 14:47:14 dist-test-slave-cr22 krb5kdc[22126](info): commencing operation
krb5kdc: starting...
W20260619 14:47:15.719802 392 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:15 dist-test-slave-cr22 krb5kdc[22126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:47:15.851423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40773
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:15.965443 22139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:15.965710 22139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:15.965785 22139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:15.969810 22139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:15.969868 22139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:15.969888 22139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:15.969904 22139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:15.969949 22139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:15.975179 22139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40773
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:15.976253 22139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:15.977236 22139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:15.985831 22145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:15.985823 22144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:15.985913 22139 server_base.cc:1034] running on GCE node
W20260619 14:47:15.985838 22147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:15.987145 22139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:15.988476 22139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:15.989686 22139 hybrid_clock.cc:648] HybridClock initialized: now 1781880435989667 us; error 34 us; skew 500 ppm
Jun 19 14:47:15 dist-test-slave-cr22 krb5kdc[22126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:15.996399 22139 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:47:15.997375 22139 webserver.cc:460] Webserver started at http://127.0.98.62:46081/ using document root <none> and password file <none>
I20260619 14:47:15.998018 22139 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:15.998111 22139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:15.998351 22139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:16.000653 22139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "b8250b49762149a993797e2bf2ce476d"
format_stamp: "Formatted at 2026-06-19 14:47:15 on dist-test-slave-cr22"
server_key: "f5791e0159c150f41e357037a3d2fbe4"
server_key_iv: "3a8bc8727f55037af010e85391651209"
server_key_version: "encryptionkey@0"
I20260619 14:47:16.001469 22139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "b8250b49762149a993797e2bf2ce476d"
format_stamp: "Formatted at 2026-06-19 14:47:15 on dist-test-slave-cr22"
server_key: "f5791e0159c150f41e357037a3d2fbe4"
server_key_iv: "3a8bc8727f55037af010e85391651209"
server_key_version: "encryptionkey@0"
I20260619 14:47:16.005770 22139 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:47:16.008634 22154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:16.010083 22139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:16.010251 22139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "b8250b49762149a993797e2bf2ce476d"
format_stamp: "Formatted at 2026-06-19 14:47:15 on dist-test-slave-cr22"
server_key: "f5791e0159c150f41e357037a3d2fbe4"
server_key_iv: "3a8bc8727f55037af010e85391651209"
server_key_version: "encryptionkey@0"
I20260619 14:47:16.010385 22139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:16.036774 22139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:16.040428 22139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:16.040647 22139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:16.048203 22139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40773
I20260619 14:47:16.048259 22206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40773 every 8 connection(s)
I20260619 14:47:16.049306 22139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:16.052101 22207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:16.057942 22207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d: Bootstrap starting.
I20260619 14:47:16.058876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 22139
I20260619 14:47:16.059041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:16.059386 392 external_mini_cluster.cc:1428] Setting key df53342b73eb7ade341f5a1d89f8d1ce
I20260619 14:47:16.061420 22207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:16.062372 22207 log.cc:824] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:16.064761 22207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d: No bootstrap required, opened a new log
I20260619 14:47:16.069020 22207 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8250b49762149a993797e2bf2ce476d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40773 } }
I20260619 14:47:16.069325 22207 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:16.069427 22207 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8250b49762149a993797e2bf2ce476d, State: Initialized, Role: FOLLOWER
I20260619 14:47:16.069983 22207 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8250b49762149a993797e2bf2ce476d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40773 } }
I20260619 14:47:16.070156 22207 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:16.070221 22207 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:16.070314 22207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:16.070732 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:16.060826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36883 (local address 127.0.98.62:40773)
0619 14:47:16.061719 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:16.061741 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:16.061791 (+ 50us) server_negotiation.cc:407] Connection header received
0619 14:47:16.062727 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:16.062754 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:16.063108 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:16.063550 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:16.063956 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:16.065431 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:16.066578 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:16.067320 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:16.067563 (+ 243us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:16.068518 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:16.068524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:16.068542 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:16.069045 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:16.069121 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:16.069152 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:16.069285 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:16.069597 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 14:47:16.069736 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":509,"threads_started":1}
I20260619 14:47:16.071482 22207 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8250b49762149a993797e2bf2ce476d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40773 } }
I20260619 14:47:16.071838 22207 leader_election.cc:302] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8250b49762149a993797e2bf2ce476d; no voters:
I20260619 14:47:16.072156 22207 leader_election.cc:288] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:16.072288 22212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:16.072540 22212 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d [term 1 LEADER]: Becoming Leader. State: Replica: b8250b49762149a993797e2bf2ce476d, State: Running, Role: LEADER
I20260619 14:47:16.073207 22212 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8250b49762149a993797e2bf2ce476d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40773 } }
I20260619 14:47:16.073501 22207 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:16.073992 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:47:16.074476 22212 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8250b49762149a993797e2bf2ce476d. Latest consensus state: current_term: 1 leader_uuid: "b8250b49762149a993797e2bf2ce476d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8250b49762149a993797e2bf2ce476d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40773 } } }
I20260619 14:47:16.074594 22212 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:16.076352 22213 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8250b49762149a993797e2bf2ce476d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8250b49762149a993797e2bf2ce476d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40773 } } }
I20260619 14:47:16.077093 22213 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d [sys.catalog]: This master's current role is: LEADER
W20260619 14:47:16.080282 22231 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:47:16.080370 22231 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:47:16.080432 22221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:16.081413 22221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:16.081773 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:16.076866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33465 (local address 127.0.98.62:40773)
0619 14:47:16.077037 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:16.077043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:16.077171 (+ 128us) server_negotiation.cc:407] Connection header received
0619 14:47:16.077409 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:16.077411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:16.077455 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:16.077538 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:16.077858 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:16.078414 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:16.079677 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:16.080103 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:16.080218 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:16.081198 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:16.081202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:16.081206 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:16.081220 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:16.081250 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:16.081263 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:16.081361 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:16.081598 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 14:47:16.081637 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 14:47:16.087080 22221 catalog_manager.cc:1347] Generated new cluster ID: c7486f036fc543e2aecaed5b1ed92762
I20260619 14:47:16.087141 22221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:16.101853 22221 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:16.103199 22221 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:16.125119 22221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8250b49762149a993797e2bf2ce476d: Generated new TSK 0
I20260619 14:47:16.126036 22221 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:47:16.134589 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 22139
W20260619 14:47:16.139235 22124 connection.cc:535] client connection to 127.0.98.62:40773 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40773): Connection reset by peer (error 104)
2026-06-19T14:47:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1504 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:16 dist-test-slave-cr22 krb5kdc[22238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:16 dist-test-slave-cr22 krb5kdc[22238](info): set up 2 sockets
Jun 19 14:47:16 dist-test-slave-cr22 krb5kdc[22238](info): commencing operation
krb5kdc: starting...
W20260619 14:47:17.207829 392 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:17 dist-test-slave-cr22 krb5kdc[22238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:47:17.342686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42027
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:17.463044 22251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:17.463384 22251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:17.463506 22251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:17.467677 22251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:17.467767 22251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:17.467814 22251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:17.467856 22251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:17.467904 22251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:17.473464 22251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42027
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:17.474655 22251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:17.475755 22251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:17.484688 22259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:17.484688 22256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:17.484683 22257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:17.484936 22251 server_base.cc:1034] running on GCE node
I20260619 14:47:17.485638 22251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:17.486773 22251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:17.488047 22251 hybrid_clock.cc:648] HybridClock initialized: now 1781880437488056 us; error 94 us; skew 500 ppm
Jun 19 14:47:17 dist-test-slave-cr22 krb5kdc[22238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:17.491216 22251 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:47:17.492195 22251 webserver.cc:460] Webserver started at http://127.0.98.62:44419/ using document root <none> and password file <none>
I20260619 14:47:17.492862 22251 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:17.492970 22251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:17.493350 22251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:17.495934 22251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "a4b86d0a710f4a7097dba0a1e9d74b9e"
format_stamp: "Formatted at 2026-06-19 14:47:17 on dist-test-slave-cr22"
server_key: "5fa0cdd5bc7faee6b1489758ff7dfa46"
server_key_iv: "dbbcdc534e968bc982099e34aaf7cda5"
server_key_version: "encryptionkey@0"
I20260619 14:47:17.496575 22251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4b86d0a710f4a7097dba0a1e9d74b9e"
format_stamp: "Formatted at 2026-06-19 14:47:17 on dist-test-slave-cr22"
server_key: "5fa0cdd5bc7faee6b1489758ff7dfa46"
server_key_iv: "dbbcdc534e968bc982099e34aaf7cda5"
server_key_version: "encryptionkey@0"
I20260619 14:47:17.501255 22251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 14:47:17.504150 22266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:17.505704 22251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:17.505867 22251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "a4b86d0a710f4a7097dba0a1e9d74b9e"
format_stamp: "Formatted at 2026-06-19 14:47:17 on dist-test-slave-cr22"
server_key: "5fa0cdd5bc7faee6b1489758ff7dfa46"
server_key_iv: "dbbcdc534e968bc982099e34aaf7cda5"
server_key_version: "encryptionkey@0"
I20260619 14:47:17.506007 22251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:17.512104 22251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:17.515997 22251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:17.516219 22251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:17.524915 22251 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42027
I20260619 14:47:17.525013 22318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42027 every 8 connection(s)
I20260619 14:47:17.526005 22251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:17.529026 22319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:17.530295 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 22251
I20260619 14:47:17.530467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:17.530777 392 external_mini_cluster.cc:1428] Setting key 758ae7ff965584cc9b62bd72d557d06c
I20260619 14:47:17.535856 22319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e: Bootstrap starting.
I20260619 14:47:17.537108 22322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:17.532354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49903 (local address 127.0.98.62:42027)
0619 14:47:17.533115 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:17.533130 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:17.533430 (+ 300us) server_negotiation.cc:407] Connection header received
0619 14:47:17.534132 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:17.534158 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:17.534455 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:17.534834 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:17.535700 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:17.535704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:17.535716 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:17.535726 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:17.535779 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:17.535798 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:17.535965 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:17.536036 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 14:47:17.536320 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":283,"threads_started":1}
I20260619 14:47:17.538419 22319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:17.539359 22319 log.cc:824] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:17.540531 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:47:17.541975 22319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e: No bootstrap required, opened a new log
I20260619 14:47:17.544766 22322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:17.542959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43107 (local address 127.0.98.62:42027)
0619 14:47:17.543089 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:17.543094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:17.543266 (+ 172us) server_negotiation.cc:407] Connection header received
0619 14:47:17.543367 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:17.543370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:17.543405 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:17.543528 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:17.544326 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:17.544329 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:17.544332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:17.544343 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:17.544370 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:17.544389 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:17.544480 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:17.544603 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 14:47:17.544642 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:47:17.546193 22319 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b86d0a710f4a7097dba0a1e9d74b9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42027 } }
I20260619 14:47:17.546422 22319 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:17.546492 22319 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b86d0a710f4a7097dba0a1e9d74b9e, State: Initialized, Role: FOLLOWER
I20260619 14:47:17.546947 22319 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b86d0a710f4a7097dba0a1e9d74b9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42027 } }
I20260619 14:47:17.547087 22319 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:17.547171 22319 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:17.547286 22319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:17.548373 22319 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b86d0a710f4a7097dba0a1e9d74b9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42027 } }
I20260619 14:47:17.548864 22319 leader_election.cc:302] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4b86d0a710f4a7097dba0a1e9d74b9e; no voters:
I20260619 14:47:17.549212 22319 leader_election.cc:288] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:17.549439 22329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:17.549743 22329 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e [term 1 LEADER]: Becoming Leader. State: Replica: a4b86d0a710f4a7097dba0a1e9d74b9e, State: Running, Role: LEADER
I20260619 14:47:17.550341 22329 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b86d0a710f4a7097dba0a1e9d74b9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42027 } }
I20260619 14:47:17.550542 22319 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:17.551975 22330 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4b86d0a710f4a7097dba0a1e9d74b9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b86d0a710f4a7097dba0a1e9d74b9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42027 } } }
I20260619 14:47:17.552124 22330 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:17.552477 22331 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4b86d0a710f4a7097dba0a1e9d74b9e. Latest consensus state: current_term: 1 leader_uuid: "a4b86d0a710f4a7097dba0a1e9d74b9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b86d0a710f4a7097dba0a1e9d74b9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42027 } } }
I20260619 14:47:17.552568 22331 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:17.552955 22338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:17.555864 22338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:17.562994 22338 catalog_manager.cc:1347] Generated new cluster ID: 164f9a64f3814a03aaea74c9a31b11e7
I20260619 14:47:17.563093 22338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:17.595009 22338 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:17.595391 22338 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:17.600292 22338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4b86d0a710f4a7097dba0a1e9d74b9e: Generated new TSK 0
I20260619 14:47:17.601140 22338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:47:17.613322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 22251
2026-06-19T14:47:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1480 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:17 dist-test-slave-cr22 krb5kdc[22350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:17 dist-test-slave-cr22 krb5kdc[22350](info): set up 2 sockets
Jun 19 14:47:17 dist-test-slave-cr22 krb5kdc[22350](info): commencing operation
krb5kdc: starting...
W20260619 14:47:18.679852 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:18 dist-test-slave-cr22 krb5kdc[22350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:47:18.806648 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40817
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:18.924226 22363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:18.924568 22363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:18.924654 22363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:18.928885 22363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:18.928994 22363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:18.929034 22363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:18.929097 22363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:18.929136 22363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:18.935073 22363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:18.936342 22363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:18.937407 22363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:18.946461 22371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:18.946623 22363 server_base.cc:1034] running on GCE node
W20260619 14:47:18.946669 22368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:18.946822 22369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:18.947345 22363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:18.948537 22363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:18.949748 22363 hybrid_clock.cc:648] HybridClock initialized: now 1781880438949735 us; error 28 us; skew 500 ppm
Jun 19 14:47:18 dist-test-slave-cr22 krb5kdc[22350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:18.952920 22363 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:47:18.953845 22363 webserver.cc:460] Webserver started at http://127.0.98.62:37275/ using document root <none> and password file <none>
I20260619 14:47:18.954440 22363 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:18.954547 22363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:18.954788 22363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:18.956952 22363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "df667420042442beb6f5f5feb63b99bd"
format_stamp: "Formatted at 2026-06-19 14:47:18 on dist-test-slave-cr22"
server_key: "b145b983b7d6ad02b35f615aeab141f4"
server_key_iv: "d45cc8ea706eef6ac3776afc01c20a30"
server_key_version: "encryptionkey@0"
I20260619 14:47:18.957540 22363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "df667420042442beb6f5f5feb63b99bd"
format_stamp: "Formatted at 2026-06-19 14:47:18 on dist-test-slave-cr22"
server_key: "b145b983b7d6ad02b35f615aeab141f4"
server_key_iv: "d45cc8ea706eef6ac3776afc01c20a30"
server_key_version: "encryptionkey@0"
I20260619 14:47:18.961346 22363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:47:18.963878 22378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:18.965276 22363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:18.965435 22363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "df667420042442beb6f5f5feb63b99bd"
format_stamp: "Formatted at 2026-06-19 14:47:18 on dist-test-slave-cr22"
server_key: "b145b983b7d6ad02b35f615aeab141f4"
server_key_iv: "d45cc8ea706eef6ac3776afc01c20a30"
server_key_version: "encryptionkey@0"
I20260619 14:47:18.965579 22363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:18.985051 22363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:18.988111 22363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:18.988353 22363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:18.995497 22363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40817
I20260619 14:47:18.995517 22430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40817 every 8 connection(s)
I20260619 14:47:18.996479 22363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:18.999387 22431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:19.003221 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 22363
I20260619 14:47:19.003332 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:19.003659 392 external_mini_cluster.cc:1428] Setting key 9b6f93a99dfc872899754b70c09b6bde
I20260619 14:47:19.005503 22431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd: Bootstrap starting.
I20260619 14:47:19.008288 22431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:19.009179 22431 log.cc:824] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:19.011166 22431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd: No bootstrap required, opened a new log
Jun 19 14:47:19 dist-test-slave-cr22 krb5kdc[22350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:47:19.014953 22431 raft_consensus.cc:348] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df667420042442beb6f5f5feb63b99bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40817 } }
I20260619 14:47:19.015201 22431 raft_consensus.cc:374] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:19.015282 22431 raft_consensus.cc:729] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df667420042442beb6f5f5feb63b99bd, State: Initialized, Role: FOLLOWER
I20260619 14:47:19.015821 22431 consensus_queue.cc:260] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df667420042442beb6f5f5feb63b99bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40817 } }
I20260619 14:47:19.015995 22431 raft_consensus.cc:388] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:19.016093 22431 raft_consensus.cc:482] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:19.016191 22431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:19.017096 22431 raft_consensus.cc:504] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df667420042442beb6f5f5feb63b99bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40817 } }
I20260619 14:47:19.017438 22431 leader_election.cc:302] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df667420042442beb6f5f5feb63b99bd; no voters:
I20260619 14:47:19.017834 22431 leader_election.cc:288] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:19.017946 22436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:19.018132 22436 raft_consensus.cc:686] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd [term 1 LEADER]: Becoming Leader. State: Replica: df667420042442beb6f5f5feb63b99bd, State: Running, Role: LEADER
I20260619 14:47:19.018518 22436 consensus_queue.cc:237] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df667420042442beb6f5f5feb63b99bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40817 } }
I20260619 14:47:19.018904 22434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:19.005152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46127 (local address 127.0.98.62:40817)
0619 14:47:19.005603 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:19.005622 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:19.005674 (+ 52us) server_negotiation.cc:407] Connection header received
0619 14:47:19.006553 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:19.006576 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:19.007039 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:19.007394 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:19.008149 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:19.009689 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:19.010581 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:19.011056 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:19.011240 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:19.013799 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:19.013804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:19.013817 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:19.013843 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:19.015732 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:19.016238 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:19.016244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:19.016249 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:19.016410 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:19.016701 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:19.016705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:19.016707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:19.017115 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:19.017330 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:19.017572 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 14:47:19.017895 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":122,"threads_started":1}
I20260619 14:47:19.019093 22431 sys_catalog.cc:564] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:19.019724 22438 sys_catalog.cc:455] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader df667420042442beb6f5f5feb63b99bd. Latest consensus state: current_term: 1 leader_uuid: "df667420042442beb6f5f5feb63b99bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df667420042442beb6f5f5feb63b99bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40817 } } }
I20260619 14:47:19.019903 22438 sys_catalog.cc:458] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:19.020169 22437 sys_catalog.cc:455] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df667420042442beb6f5f5feb63b99bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df667420042442beb6f5f5feb63b99bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40817 } } }
I20260619 14:47:19.020267 22437 sys_catalog.cc:458] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:19.020454 22443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:19.022258 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:47:19.024019 22443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:19.030833 22443 catalog_manager.cc:1347] Generated new cluster ID: 13c791a3315349079564a1bb2cc0d54f
I20260619 14:47:19.030910 22443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:19.033579 22434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:19.024660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40745 (local address 127.0.98.62:40817)
0619 14:47:19.024762 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:19.024765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:19.024778 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:47:19.024814 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:19.024817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:19.024859 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:19.024944 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:19.025758 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:19.026526 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:19.027577 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:19.027897 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:19.028008 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:19.029716 (+ 1708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:19.029720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:19.029723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:19.029748 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:19.031794 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:19.032297 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:19.032301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:19.032303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:19.032394 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:19.032681 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:19.032685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:19.032689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:19.032911 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:19.033047 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:19.033268 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 14:47:19.033403 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:47:19.062517 22443 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:19.063565 22443 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:19.072485 22443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df667420042442beb6f5f5feb63b99bd: Generated new TSK 0
I20260619 14:47:19.073320 22443 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:47:19.084396 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 22363
2026-06-19T14:47:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1471 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:19 dist-test-slave-cr22 krb5kdc[22462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:19 dist-test-slave-cr22 krb5kdc[22462](info): set up 2 sockets
Jun 19 14:47:19 dist-test-slave-cr22 krb5kdc[22462](info): commencing operation
krb5kdc: starting...
W20260619 14:47:20.144155 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:20 dist-test-slave-cr22 krb5kdc[22462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:47:20.278538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36041
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:20.388978 22475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:20.389303 22475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:20.389381 22475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:20.393450 22475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:20.393554 22475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:20.393595 22475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:20.393654 22475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:20.393689 22475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:20.399111 22475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:20.400274 22475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:20.401440 22475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:47:20.410408 22475 server_base.cc:1034] running on GCE node
W20260619 14:47:20.410390 22481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:20.410390 22480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:20.410624 22483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:20.411408 22475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:20.412627 22475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:20.413837 22475 hybrid_clock.cc:648] HybridClock initialized: now 1781880440413814 us; error 34 us; skew 500 ppm
Jun 19 14:47:20 dist-test-slave-cr22 krb5kdc[22462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:20.416829 22475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:47:20.417817 22475 webserver.cc:460] Webserver started at http://127.0.98.62:39645/ using document root <none> and password file <none>
I20260619 14:47:20.418512 22475 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:20.418624 22475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:20.418970 22475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:20.421165 22475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "98b61e0c0551415e9685e981b80470b6"
format_stamp: "Formatted at 2026-06-19 14:47:20 on dist-test-slave-cr22"
server_key: "f72ee04a871abb0a73a87195e457307f"
server_key_iv: "5e229859c7eb1104638f2bdc56b88b5a"
server_key_version: "encryptionkey@0"
I20260619 14:47:20.421751 22475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "98b61e0c0551415e9685e981b80470b6"
format_stamp: "Formatted at 2026-06-19 14:47:20 on dist-test-slave-cr22"
server_key: "f72ee04a871abb0a73a87195e457307f"
server_key_iv: "5e229859c7eb1104638f2bdc56b88b5a"
server_key_version: "encryptionkey@0"
I20260619 14:47:20.425843 22475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:47:20.428483 22490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:20.429733 22475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 14:47:20.429878 22475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "98b61e0c0551415e9685e981b80470b6"
format_stamp: "Formatted at 2026-06-19 14:47:20 on dist-test-slave-cr22"
server_key: "f72ee04a871abb0a73a87195e457307f"
server_key_iv: "5e229859c7eb1104638f2bdc56b88b5a"
server_key_version: "encryptionkey@0"
I20260619 14:47:20.430001 22475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:20.467738 22475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:20.471190 22475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:20.471455 22475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:20.478902 22475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36041
I20260619 14:47:20.478937 22542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36041 every 8 connection(s)
I20260619 14:47:20.479913 22475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:20.482780 22543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:20.485630 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 22475
I20260619 14:47:20.485778 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:20.486048 392 external_mini_cluster.cc:1428] Setting key dd04ca60ad30912059825bbfce7d1a55
I20260619 14:47:20.488772 22543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6: Bootstrap starting.
I20260619 14:47:20.491034 22543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:20.491853 22543 log.cc:824] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:20.494576 22543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6: No bootstrap required, opened a new log
I20260619 14:47:20.496387 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:20.487504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40753 (local address 127.0.98.62:36041)
0619 14:47:20.488034 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:20.488055 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:20.488095 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:47:20.488792 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:20.488810 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:20.489281 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:20.489653 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:20.490100 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:20.491189 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:20.492330 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:20.493215 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:20.493463 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:20.494494 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:20.494499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:20.494513 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:20.494988 (+ 475us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:20.495054 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:20.495079 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:20.495202 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:20.495331 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 14:47:20.495595 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":165,"threads_started":1}
I20260619 14:47:20.498972 22543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b61e0c0551415e9685e981b80470b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36041 } }
I20260619 14:47:20.499243 22543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:20.499334 22543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98b61e0c0551415e9685e981b80470b6, State: Initialized, Role: FOLLOWER
I20260619 14:47:20.499846 22543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b61e0c0551415e9685e981b80470b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36041 } }
I20260619 14:47:20.500021 22543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:20.500114 22543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:20.500109 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:47:20.500238 22543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:20.501271 22543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b61e0c0551415e9685e981b80470b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36041 } }
I20260619 14:47:20.501641 22543 leader_election.cc:302] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98b61e0c0551415e9685e981b80470b6; no voters:
I20260619 14:47:20.501986 22543 leader_election.cc:288] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:20.502091 22552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:20.502323 22552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6 [term 1 LEADER]: Becoming Leader. State: Replica: 98b61e0c0551415e9685e981b80470b6, State: Running, Role: LEADER
I20260619 14:47:20.502787 22552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b61e0c0551415e9685e981b80470b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36041 } }
I20260619 14:47:20.503224 22543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:20.504081 22552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98b61e0c0551415e9685e981b80470b6. Latest consensus state: current_term: 1 leader_uuid: "98b61e0c0551415e9685e981b80470b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b61e0c0551415e9685e981b80470b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36041 } } }
I20260619 14:47:20.504190 22552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:20.504832 22553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98b61e0c0551415e9685e981b80470b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b61e0c0551415e9685e981b80470b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36041 } } }
I20260619 14:47:20.504944 22553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:20.505301 22562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:20.507313 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:20.502469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45669 (local address 127.0.98.62:36041)
0619 14:47:20.502587 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:20.502592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:20.502730 (+ 138us) server_negotiation.cc:407] Connection header received
0619 14:47:20.502919 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:20.502922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:20.502970 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:20.503034 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:20.503488 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:20.504246 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:20.505484 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:20.505802 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:20.505875 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:20.506874 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:20.506878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:20.506880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:20.506894 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:20.506920 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:20.506931 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:20.507021 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:20.507149 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 14:47:20.507203 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:47:20.508685 22562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:20.514289 22562 catalog_manager.cc:1347] Generated new cluster ID: 5bbee235418f4ec3a725baa1b19087d4
I20260619 14:47:20.514371 22562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:20.528862 22562 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:20.530140 22562 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:20.539750 22562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98b61e0c0551415e9685e981b80470b6: Generated new TSK 0
I20260619 14:47:20.540630 22562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:47:20.554970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 22475
2026-06-19T14:47:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1469 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:20 dist-test-slave-cr22 krb5kdc[22574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:20 dist-test-slave-cr22 krb5kdc[22574](info): set up 2 sockets
Jun 19 14:47:20 dist-test-slave-cr22 krb5kdc[22574](info): commencing operation
krb5kdc: starting...
W20260619 14:47:21.611871 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:21 dist-test-slave-cr22 krb5kdc[22574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:47:21.739423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39425
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:21.855120 22587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:21.855495 22587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:21.855587 22587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:21.859944 22587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:21.860054 22587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:21.860090 22587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:21.860129 22587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:21.860177 22587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:21.865809 22587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:21.867132 22587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:21.868232 22587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:21.877794 22592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:21.877787 22595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:21.877926 22593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:21.878626 22587 server_base.cc:1034] running on GCE node
I20260619 14:47:21.879335 22587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:21.880607 22587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:21.881834 22587 hybrid_clock.cc:648] HybridClock initialized: now 1781880441881819 us; error 28 us; skew 500 ppm
Jun 19 14:47:21 dist-test-slave-cr22 krb5kdc[22574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:21.890161 22587 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:47:21.891146 22587 webserver.cc:460] Webserver started at http://127.0.98.62:42141/ using document root <none> and password file <none>
I20260619 14:47:21.891758 22587 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:21.891824 22587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:21.891999 22587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:21.894019 22587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "a07dd1ae0c9244d3bcfca70596bc0ed1"
format_stamp: "Formatted at 2026-06-19 14:47:21 on dist-test-slave-cr22"
server_key: "44cdc161cfb6c19dc610a473c627ea1b"
server_key_iv: "e81c418f70baa92a5219bd4de821b0e7"
server_key_version: "encryptionkey@0"
I20260619 14:47:21.894531 22587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "a07dd1ae0c9244d3bcfca70596bc0ed1"
format_stamp: "Formatted at 2026-06-19 14:47:21 on dist-test-slave-cr22"
server_key: "44cdc161cfb6c19dc610a473c627ea1b"
server_key_iv: "e81c418f70baa92a5219bd4de821b0e7"
server_key_version: "encryptionkey@0"
I20260619 14:47:21.898653 22587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:47:21.901211 22602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:21.902462 22587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:47:21.902731 22587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "a07dd1ae0c9244d3bcfca70596bc0ed1"
format_stamp: "Formatted at 2026-06-19 14:47:21 on dist-test-slave-cr22"
server_key: "44cdc161cfb6c19dc610a473c627ea1b"
server_key_iv: "e81c418f70baa92a5219bd4de821b0e7"
server_key_version: "encryptionkey@0"
I20260619 14:47:21.902839 22587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:21.907672 22587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:21.910786 22587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:21.910996 22587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:21.918430 22587 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39425
I20260619 14:47:21.918458 22654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39425 every 8 connection(s)
I20260619 14:47:21.919405 22587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:21.922309 22655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:21.926285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 22587
I20260619 14:47:21.926409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:21.926714 392 external_mini_cluster.cc:1428] Setting key 6ee7eb4be59cebb7ec3a8e59ec0dc031
I20260619 14:47:21.928427 22655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1: Bootstrap starting.
I20260619 14:47:21.930824 22655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:21.931674 22655 log.cc:824] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:21.932425 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:21.928463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39553 (local address 127.0.98.62:39425)
0619 14:47:21.929022 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:21.929038 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:21.929076 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:47:21.929717 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:21.929735 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:21.930088 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:21.930459 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:21.931193 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:21.931198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:21.931209 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:21.931218 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:21.931266 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:21.931285 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:21.931401 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:21.931563 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 14:47:21.931721 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":150,"threads_started":1}
I20260619 14:47:21.933525 22655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1: No bootstrap required, opened a new log
I20260619 14:47:21.935259 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:47:21.937574 22655 raft_consensus.cc:348] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07dd1ae0c9244d3bcfca70596bc0ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } }
I20260619 14:47:21.937824 22655 raft_consensus.cc:374] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:21.937911 22655 raft_consensus.cc:729] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07dd1ae0c9244d3bcfca70596bc0ed1, State: Initialized, Role: FOLLOWER
I20260619 14:47:21.938416 22655 consensus_queue.cc:260] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07dd1ae0c9244d3bcfca70596bc0ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } }
I20260619 14:47:21.938601 22655 raft_consensus.cc:388] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:21.938690 22655 raft_consensus.cc:482] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:21.938807 22655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:21.939225 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:21.937445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37899 (local address 127.0.98.62:39425)
0619 14:47:21.937554 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:21.937559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:21.937780 (+ 221us) server_negotiation.cc:407] Connection header received
0619 14:47:21.937952 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:21.937954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:21.937992 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:21.938065 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:21.938857 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:21.938863 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:21.938865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:21.938879 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:21.938908 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:21.938921 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:21.939014 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:21.939069 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 14:47:21.939109 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 14:47:21.939977 22655 raft_consensus.cc:504] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07dd1ae0c9244d3bcfca70596bc0ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } }
I20260619 14:47:21.940411 22655 leader_election.cc:302] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a07dd1ae0c9244d3bcfca70596bc0ed1; no voters:
I20260619 14:47:21.940416 22665 raft_consensus.cc:482] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:47:21.940547 22665 raft_consensus.cc:504] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07dd1ae0c9244d3bcfca70596bc0ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } }
I20260619 14:47:21.940748 22655 leader_election.cc:288] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:21.940853 22666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:21.941069 22665 leader_election.cc:302] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a07dd1ae0c9244d3bcfca70596bc0ed1; no voters:
I20260619 14:47:21.941078 22666 raft_consensus.cc:686] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [term 1 LEADER]: Becoming Leader. State: Replica: a07dd1ae0c9244d3bcfca70596bc0ed1, State: Running, Role: LEADER
I20260619 14:47:21.941298 22665 leader_election.cc:288] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers
I20260619 14:47:21.941437 22666 consensus_queue.cc:237] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07dd1ae0c9244d3bcfca70596bc0ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } }
I20260619 14:47:21.942019 22655 sys_catalog.cc:564] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:21.943244 22667 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a07dd1ae0c9244d3bcfca70596bc0ed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07dd1ae0c9244d3bcfca70596bc0ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } } }
I20260619 14:47:21.943589 22667 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:21.943614 22669 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a07dd1ae0c9244d3bcfca70596bc0ed1. Latest consensus state: current_term: 1 leader_uuid: "a07dd1ae0c9244d3bcfca70596bc0ed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07dd1ae0c9244d3bcfca70596bc0ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } } }
I20260619 14:47:21.943734 22669 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:21.944332 22679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:21.947309 22679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:21.952917 22679 catalog_manager.cc:1347] Generated new cluster ID: 99afe4f7a87d43a59560caf204c422ee
I20260619 14:47:21.953064 22679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:21.968786 22679 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:21.969127 22679 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:21.981194 22679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a07dd1ae0c9244d3bcfca70596bc0ed1: Generated new TSK 0
I20260619 14:47:21.981933 22679 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:47:21.993323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 22587
2026-06-19T14:47:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1440 ms)
[----------] 18 tests from AuthTokenIssuingTest (26748 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:22 dist-test-slave-cr22 krb5kdc[22688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:22 dist-test-slave-cr22 krb5kdc[22688](info): set up 2 sockets
Jun 19 14:47:22 dist-test-slave-cr22 krb5kdc[22688](info): commencing operation
krb5kdc: starting...
W20260619 14:47:23.104336 392 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:23 dist-test-slave-cr22 krb5kdc[22688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:47:23.235774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40633
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40633,127.0.98.61:40755,127.0.98.60:33737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:23.349200 22701 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:47:23.349562 22701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:23.349646 22701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:23.349737 22701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:23.353946 22701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:23.354053 22701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:23.354092 22701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:23.354130 22701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:23.354166 22701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:23.359652 22701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40633,127.0.98.61:40755,127.0.98.60:33737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:23.360800 22701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:23.361892 22701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:47:23.371202 22701 server_base.cc:1034] running on GCE node
W20260619 14:47:23.371295 22706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:23.371176 22707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:23.371562 22709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:23.372143 22701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:23.373320 22701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:23.374516 22701 hybrid_clock.cc:648] HybridClock initialized: now 1781880443374500 us; error 31 us; skew 500 ppm
Jun 19 14:47:23 dist-test-slave-cr22 krb5kdc[22688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:23.377663 22701 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:47:23.378545 22701 webserver.cc:460] Webserver started at http://127.0.98.62:36795/ using document root <none> and password file <none>
I20260619 14:47:23.379108 22701 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:23.379202 22701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:23.379495 22701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:23.381557 22701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "2c6981a7854d482589ec0107fd23a7d9"
format_stamp: "Formatted at 2026-06-19 14:47:23 on dist-test-slave-cr22"
server_key: "d933b2e17caaa1738d643ab9b5b9c0a8"
server_key_iv: "044854fe8b4ec4af4d2e7f278fa56f45"
server_key_version: "encryptionkey@0"
I20260619 14:47:23.382172 22701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c6981a7854d482589ec0107fd23a7d9"
format_stamp: "Formatted at 2026-06-19 14:47:23 on dist-test-slave-cr22"
server_key: "d933b2e17caaa1738d643ab9b5b9c0a8"
server_key_iv: "044854fe8b4ec4af4d2e7f278fa56f45"
server_key_version: "encryptionkey@0"
I20260619 14:47:23.385991 22701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 14:47:23.388808 22716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:23.390372 22701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:47:23.390554 22701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "2c6981a7854d482589ec0107fd23a7d9"
format_stamp: "Formatted at 2026-06-19 14:47:23 on dist-test-slave-cr22"
server_key: "d933b2e17caaa1738d643ab9b5b9c0a8"
server_key_iv: "044854fe8b4ec4af4d2e7f278fa56f45"
server_key_version: "encryptionkey@0"
I20260619 14:47:23.390693 22701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:23.406526 22701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:23.409914 22701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:23.410161 22701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:23.418198 22701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40633
I20260619 14:47:23.418622 22768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40633 every 8 connection(s)
I20260619 14:47:23.419299 22701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:23.422366 22769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:23.422699 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 22701
I20260619 14:47:23.422858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:23.423173 392 external_mini_cluster.cc:1428] Setting key f31998cb56808b59a74e10939f93ea82
I20260619 14:47:23.427237 22769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 14:47:23 dist-test-slave-cr22 krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:47:23.443634 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.429754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40633 (local address 127.0.0.1:37558)
0619 14:47:23.430362 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.430399 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:23.431474 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:23.432208 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:23.432213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:23.432655 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:23.432921 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.432934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.434348 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.434351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.435161 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.435169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.435875 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.435879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.436003 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.436592 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:23.436612 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:23.438458 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:23.440551 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:23.440561 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:23.440570 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:23.440794 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:23.441121 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:23.441124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:23.441126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:23.441241 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:23.441851 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:23.441858 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:23.442002 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 14:47:23.442174 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 14:47:23.442597 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":121,"spinlock_wait_cycles":40192,"thread_start_us":166,"threads_started":1}
I20260619 14:47:23.443634 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.429753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37558 (local address 127.0.98.62:40633)
0619 14:47:23.430214 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:23.430234 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:23.430511 (+ 277us) server_negotiation.cc:407] Connection header received
0619 14:47:23.431824 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:23.431828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:23.432012 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:23.432211 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:23.433075 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.434195 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.435303 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.435723 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.435916 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.438614 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:23.438619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:23.438628 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:23.438647 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:23.440408 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.440904 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.440909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.440911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.441020 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.441340 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.441342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.441344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.441727 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:23.441871 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:23.442231 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 14:47:23.442548 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":310,"thread_start_us":161,"threads_started":1}
I20260619 14:47:23.446360 22769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:47:23.447139 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.446878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40755 (local address 127.0.0.1:50894)
0619 14:47:23.446996 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.447065 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 14:47:23.447364 22718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40755: connect: Connection refused (error 111)
I20260619 14:47:23.448509 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.448303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40755 (local address 127.0.0.1:50908)
0619 14:47:23.448406 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.448456 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 14:47:23.448698 22769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40755: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40755: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 14:47:23.450109 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40755
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40633,127.0.98.61:40755,127.0.98.60:33737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:47:23.498001 22769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } attempt: 1
I20260619 14:47:23.498955 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.498618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40755 (local address 127.0.0.1:50920)
0619 14:47:23.498813 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.498876 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 14:47:23.499776 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.499552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40755 (local address 127.0.0.1:50934)
0619 14:47:23.499662 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.499717 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 14:47:23.499905 22769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40755: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40755: connect: Connection refused (error 111)
I20260619 14:47:23.568100 22769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } attempt: 2
I20260619 14:47:23.568957 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.568644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40755 (local address 127.0.0.1:50936)
0619 14:47:23.568813 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.568878 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 14:47:23.569777 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.569538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40755 (local address 127.0.0.1:50948)
0619 14:47:23.569655 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.569702 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 14:47:23.570043 22769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40755: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40755: connect: Connection refused (error 111)
W20260619 14:47:23.574466 22776 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:47:23.574760 22776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:23.574837 22776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:23.574923 22776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:23.579396 22776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:23.579532 22776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:23.579574 22776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:23.579617 22776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:23.579653 22776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:23.585708 22776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40633,127.0.98.61:40755,127.0.98.60:33737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:23.586877 22776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:23.588006 22776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:23.597251 22784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:23.597307 22781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:23.597430 22776 server_base.cc:1034] running on GCE node
W20260619 14:47:23.597355 22782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:23.598239 22776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:23.599586 22776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:23.600801 22776 hybrid_clock.cc:648] HybridClock initialized: now 1781880443600781 us; error 35 us; skew 500 ppm
Jun 19 14:47:23 dist-test-slave-cr22 krb5kdc[22688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:23.603792 22776 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 14:47:23.604769 22776 webserver.cc:460] Webserver started at http://127.0.98.61:34399/ using document root <none> and password file <none>
I20260619 14:47:23.605365 22776 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:23.605463 22776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:23.605700 22776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:23.608263 22776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-1/data/instance:
uuid: "19ae621bccf6415b8eebf59e90db0cdd"
format_stamp: "Formatted at 2026-06-19 14:47:23 on dist-test-slave-cr22"
server_key: "31b83e8bb659cfdee53cd18637be8223"
server_key_iv: "491d9a0aeb81a7cb951e995d5388695c"
server_key_version: "encryptionkey@0"
I20260619 14:47:23.608901 22776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-1/wal/instance:
uuid: "19ae621bccf6415b8eebf59e90db0cdd"
format_stamp: "Formatted at 2026-06-19 14:47:23 on dist-test-slave-cr22"
server_key: "31b83e8bb659cfdee53cd18637be8223"
server_key_iv: "491d9a0aeb81a7cb951e995d5388695c"
server_key_version: "encryptionkey@0"
I20260619 14:47:23.613328 22776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:47:23.616189 22791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:23.617550 22776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:23.617748 22776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-1/wal
uuid: "19ae621bccf6415b8eebf59e90db0cdd"
format_stamp: "Formatted at 2026-06-19 14:47:23 on dist-test-slave-cr22"
server_key: "31b83e8bb659cfdee53cd18637be8223"
server_key_iv: "491d9a0aeb81a7cb951e995d5388695c"
server_key_version: "encryptionkey@0"
I20260619 14:47:23.617879 22776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:23.637557 22776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:23.641065 22776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:23.641311 22776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:23.649276 22776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40755
I20260619 14:47:23.649451 22843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40755 every 8 connection(s)
I20260619 14:47:23.650478 22776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-1/data/info.pb
I20260619 14:47:23.653566 22844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:23.657198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 22776
I20260619 14:47:23.657316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-1/wal/instance
I20260619 14:47:23.657639 392 external_mini_cluster.cc:1428] Setting key 1b9214a19c73e5f4cf16fbac1d94a809
I20260619 14:47:23.657613 22844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:47:23.661285 22769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } attempt: 3
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 14:47:23 dist-test-slave-cr22 krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 14:47:23 dist-test-slave-cr22 krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:47:23.676811 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.660009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37574 (local address 127.0.98.62:40633)
0619 14:47:23.660191 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:23.660195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:23.661224 (+ 1029us) server_negotiation.cc:407] Connection header received
0619 14:47:23.662564 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:23.662570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:23.662634 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:23.662732 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:23.663789 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.664816 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.666144 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.666386 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.666527 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.672675 (+ 6148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:23.672681 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:23.672686 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:23.672722 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:23.674352 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.674995 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.675002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.675007 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.675102 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.675479 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.675485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.675487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.675695 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:23.675810 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:23.676482 (+ 672us) server_negotiation.cc:299] Negotiation successful
0619 14:47:23.676642 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 14:47:23.677099 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.661746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40755 (local address 127.0.0.1:50962)
0619 14:47:23.661885 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.661901 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:23.662028 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:23.662937 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:23.662941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:23.663176 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:23.663317 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.663328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.664834 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.664841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.665714 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.665725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.666680 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.666684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.666773 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.667552 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:23.667575 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:23.671743 (+ 4168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:23.674637 (+ 2894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:23.674640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:23.674642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:23.674888 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:23.675249 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:23.675255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:23.675260 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:23.675348 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:23.676371 (+ 1023us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:23.676376 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:23.676494 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 14:47:23.676617 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 14:47:23.676801 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 14:47:23.678376 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.660448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40633 (local address 127.0.0.1:37574)
0619 14:47:23.661051 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.661097 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:23.662270 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:23.662803 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:23.662812 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:23.663286 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:23.663652 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.663665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.664966 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.664970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.666024 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.666034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.666566 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.666570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.666728 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.667643 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:23.667666 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:23.672474 (+ 4808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:23.674483 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:23.674492 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:23.674502 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:23.674867 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:23.675212 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:23.675216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:23.675222 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:23.675352 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:23.675811 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:23.675821 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:23.676206 (+ 385us) client_negotiation.cc:769] Sending connection context
0619 14:47:23.676996 (+ 790us) client_negotiation.cc:240] Negotiation successful
0619 14:47:23.677314 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":85,"spinlock_wait_cycles":16768,"thread_start_us":171,"threads_started":1}
I20260619 14:47:23.678376 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.661954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.0.98.61:40755)
0619 14:47:23.662284 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:23.662297 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:23.662337 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:47:23.662492 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:23.662514 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:23.662770 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:23.662961 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:23.663465 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.664715 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.665853 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.666496 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.666736 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.671958 (+ 5222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:23.671970 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:23.671990 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:23.672022 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:23.674519 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.675020 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.675025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.675026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.675143 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.675850 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.675853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.675854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.676247 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:23.676386 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:23.676735 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 14:47:23.677363 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":221,"thread_start_us":110,"threads_started":1}
I20260619 14:47:23.679970 22844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:47:23.680779 22769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:47:23.681993 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.681762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33737 (local address 127.0.0.1:33366)
0619 14:47:23.681864 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.681916 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 14:47:23.682806 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.682520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33737 (local address 127.0.0.1:33374)
0619 14:47:23.682713 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.682757 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260619 14:47:23.682971 22769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33737: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33737: connect: Connection refused (error 111)
Jun 19 14:47:23 dist-test-slave-cr22 krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:47:23.689617 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.680493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40755 (local address 127.0.0.1:50972)
0619 14:47:23.680802 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.680814 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:23.680871 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:23.681398 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:23.681401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:23.681537 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:23.681736 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.681744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.682808 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.682814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.683649 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.683675 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.684142 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.684146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.684257 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.684740 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:23.684752 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:23.686234 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:23.688041 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:23.688047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:23.688050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:23.688237 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:23.688539 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:23.688542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:23.688543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:23.688627 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:23.689034 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:23.689037 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:23.689145 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 14:47:23.689275 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 14:47:23.689405 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":4}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 14:47:23.689640 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.680526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.0.98.61:40755)
0619 14:47:23.681127 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:23.681134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:23.681157 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:47:23.681197 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:23.681202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:23.681251 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:23.681339 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:23.682162 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.682697 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.683779 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.684036 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.684146 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.686376 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:23.686379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:23.686384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:23.686408 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:23.687941 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.688332 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.688338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.688340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.688414 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.688713 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.688715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.688717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.688884 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:23.688960 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:23.689332 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 14:47:23.689494 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":47}
I20260619 14:47:23.690637 22844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:47:23.690932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33737
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40633,127.0.98.61:40755,127.0.98.60:33737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:47:23.691643 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.691219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33737 (local address 127.0.0.1:33390)
0619 14:47:23.691400 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.691543 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260619 14:47:23.692209 22792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33737: connect: Connection refused (error 111)
I20260619 14:47:23.693230 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.692977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33737 (local address 127.0.0.1:33404)
0619 14:47:23.693096 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.693159 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 14:47:23.693392 22844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33737: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33737: connect: Connection refused (error 111)
I20260619 14:47:23.714169 22769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } attempt: 1
I20260619 14:47:23.715261 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.714782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33737 (local address 127.0.0.1:33410)
0619 14:47:23.715125 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.715184 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260619 14:47:23.716219 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.715927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33737 (local address 127.0.0.1:33424)
0619 14:47:23.716095 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.716155 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260619 14:47:23.716435 22769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33737: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33737: connect: Connection refused (error 111)
I20260619 14:47:23.742681 22844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } attempt: 1
I20260619 14:47:23.743979 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.743599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33737 (local address 127.0.0.1:33440)
0619 14:47:23.743774 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.743860 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 14:47:23.744861 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.744599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33737 (local address 127.0.0.1:33452)
0619 14:47:23.744738 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.744794 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 14:47:23.745091 22844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33737: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33737: connect: Connection refused (error 111)
I20260619 14:47:23.791608 22769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } attempt: 2
I20260619 14:47:23.792658 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.792251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33737 (local address 127.0.0.1:33460)
0619 14:47:23.792401 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.792507 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 14:47:23.793465 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.793241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33737 (local address 127.0.0.1:33470)
0619 14:47:23.793358 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.793412 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 14:47:23.793602 22769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33737: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33737: connect: Connection refused (error 111)
W20260619 14:47:23.808537 22850 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:47:23.808866 22850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:23.808889 22850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:23.808974 22850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:23.813199 22850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:23.813313 22850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:23.813342 22850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:23.813364 22850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
I20260619 14:47:23.813275 22844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } attempt: 2
W20260619 14:47:23.813385 22850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:23.814136 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.813803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33737 (local address 127.0.0.1:33474)
0619 14:47:23.813973 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.814044 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 14:47:23.814915 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.814683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33737 (local address 127.0.0.1:33488)
0619 14:47:23.814803 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.814852 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 14:47:23.815135 22844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33737: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33737: connect: Connection refused (error 111)
I20260619 14:47:23.819203 22850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40633,127.0.98.61:40755,127.0.98.60:33737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:23.820305 22850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:23.821261 22850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:47:23.831100 22850 server_base.cc:1034] running on GCE node
W20260619 14:47:23.831295 22857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:23.831312 22856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:23.831501 22859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:23.832109 22850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:23.833276 22850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:23.834520 22850 hybrid_clock.cc:648] HybridClock initialized: now 1781880443834476 us; error 61 us; skew 500 ppm
Jun 19 14:47:23 dist-test-slave-cr22 krb5kdc[22688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:23.837723 22850 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 14:47:23.838788 22850 webserver.cc:460] Webserver started at http://127.0.98.60:33341/ using document root <none> and password file <none>
I20260619 14:47:23.839406 22850 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:23.839588 22850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:23.839845 22850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:23.842120 22850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-2/data/instance:
uuid: "9953595f091d4793884be979536b31e0"
format_stamp: "Formatted at 2026-06-19 14:47:23 on dist-test-slave-cr22"
server_key: "9eee41e34859827b4fb2a8c5ec3e63f2"
server_key_iv: "0f028babb68afefb11e7c60b49777042"
server_key_version: "encryptionkey@0"
I20260619 14:47:23.842736 22850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-2/wal/instance:
uuid: "9953595f091d4793884be979536b31e0"
format_stamp: "Formatted at 2026-06-19 14:47:23 on dist-test-slave-cr22"
server_key: "9eee41e34859827b4fb2a8c5ec3e63f2"
server_key_iv: "0f028babb68afefb11e7c60b49777042"
server_key_version: "encryptionkey@0"
I20260619 14:47:23.846869 22850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:47:23.849655 22866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:23.850991 22850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:47:23.851243 22850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-2/wal
uuid: "9953595f091d4793884be979536b31e0"
format_stamp: "Formatted at 2026-06-19 14:47:23 on dist-test-slave-cr22"
server_key: "9eee41e34859827b4fb2a8c5ec3e63f2"
server_key_iv: "0f028babb68afefb11e7c60b49777042"
server_key_version: "encryptionkey@0"
I20260619 14:47:23.851383 22850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:23.870893 22850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:23.874402 22850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:23.874629 22850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:23.882822 22850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33737
I20260619 14:47:23.883188 22918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33737 every 8 connection(s)
I20260619 14:47:23.883875 22850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-2/data/info.pb
I20260619 14:47:23.886638 22919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:23.888268 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 22850
I20260619 14:47:23.888394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880155260304-392-0/minicluster-data/master-2/wal/instance
I20260619 14:47:23.888690 392 external_mini_cluster.cc:1428] Setting key b4c46bc96273a851659882efc61449d8
I20260619 14:47:23.890803 22919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:47:23.892765 22769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } attempt: 3
Jun 19 14:47:23 dist-test-slave-cr22 krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 14:47:23 dist-test-slave-cr22 krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 14:47:23 dist-test-slave-cr22 krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:47:23.904116 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.893244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33737 (local address 127.0.0.1:33512)
0619 14:47:23.893512 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.893528 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:23.893604 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:23.894302 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:23.894307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:23.894531 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:23.894614 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.894620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.895647 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.895652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.896308 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.896321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.896994 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.897007 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.897097 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.897440 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:23.897450 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:23.898947 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:23.901528 (+ 2581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:23.901531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:23.901534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:23.901874 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:23.902284 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:23.902289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:23.902294 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:23.902390 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:23.903534 (+ 1144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:23.903537 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:23.903642 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 14:47:23.903798 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 14:47:23.903941 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"spinlock_wait_cycles":11264}
I20260619 14:47:23.904616 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.893984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37580 (local address 127.0.98.62:40633)
0619 14:47:23.894094 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:23.894098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:23.894112 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:47:23.894204 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:23.894210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:23.894279 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:23.894350 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:23.895177 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.895942 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.897782 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.898080 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.898197 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.900803 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:23.900807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:23.900809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:23.900840 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:23.902536 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.903081 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.903085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.903087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.903170 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.903493 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.903497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.903499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.903733 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:23.903838 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:23.904352 (+ 514us) server_negotiation.cc:299] Negotiation successful
0619 14:47:23.904469 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:47:23.905319 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.893281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33512 (local address 127.0.98.60:33737)
0619 14:47:23.894084 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:23.894092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:23.894105 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:47:23.894136 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:23.894139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:23.894174 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:23.894261 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:23.894733 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.895409 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.896422 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.896873 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.896991 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.899095 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:23.899102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:23.899115 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:23.899141 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:23.901380 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.902001 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.902008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.902016 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.902169 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.902495 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.902498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.902500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.902971 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:23.903095 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:23.903950 (+ 855us) server_negotiation.cc:299] Negotiation successful
0619 14:47:23.904275 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":744,"spinlock_wait_cycles":2816,"thread_start_us":143,"threads_started":1}
I20260619 14:47:23.905323 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.893240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40633 (local address 127.0.0.1:37580)
0619 14:47:23.893826 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.893861 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:23.894117 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:23.894455 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:23.894461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:23.894906 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:23.895044 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.895051 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.896053 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.896057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.897642 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.897656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.898236 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.898240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.898354 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.898839 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:23.898854 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:23.900657 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:23.902660 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:23.902668 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:23.902675 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:23.902976 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:23.903261 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:23.903266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:23.903268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:23.903363 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:23.903888 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:23.903895 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:23.904122 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 14:47:23.904733 (+ 611us) client_negotiation.cc:240] Negotiation successful
0619 14:47:23.904895 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"spinlock_wait_cycles":2432,"thread_start_us":86,"threads_started":1}
I20260619 14:47:23.905323 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.890055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.0.98.60:33737)
0619 14:47:23.890450 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:23.890471 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:23.890540 (+ 69us) server_negotiation.cc:407] Connection header received
0619 14:47:23.891252 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:23.891269 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:23.891671 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:23.891979 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:23.892886 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.894381 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.895628 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.896474 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.896618 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.899811 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:23.899815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:23.899817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:23.899844 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:23.902363 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.902899 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.902903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.902904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.902976 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.903575 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.903579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.903580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.903751 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:23.904476 (+ 725us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:23.904559 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 14:47:23.904666 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":143,"threads_started":1}
I20260619 14:47:23.906276 22844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } attempt: 3
I20260619 14:47:23.906847 22919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:47:23.909904 22769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9: Bootstrap starting.
Jun 19 14:47:23 dist-test-slave-cr22 krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:47:23.912778 22769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:23.913841 22769 log.cc:824] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:23.916752 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.907484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40755 (local address 127.0.0.1:50982)
0619 14:47:23.907572 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.907586 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:23.907658 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:23.908241 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:23.908244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:23.908412 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:23.908530 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.908537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.909365 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.909368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.910111 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.910121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.910636 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.910640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.910735 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.911342 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:23.911356 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:23.912946 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:23.914826 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:23.914831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:23.914833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:23.915103 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:23.915502 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:23.915505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:23.915507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:23.915594 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:23.916093 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:23.916096 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:23.916187 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 14:47:23.916455 (+ 268us) client_negotiation.cc:240] Negotiation successful
0619 14:47:23.916593 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 14:47:23.916847 22769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9: No bootstrap required, opened a new log
Jun 19 14:47:23 dist-test-slave-cr22 krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 14:47:23 dist-test-slave-cr22 krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:47:23.918489 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.907557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.0.98.61:40755)
0619 14:47:23.907677 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:23.907681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:23.907715 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:47:23.907992 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:23.907995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:23.908046 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:23.908129 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:23.908706 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.909246 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.910227 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.910521 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.910605 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.913116 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:23.913120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:23.913122 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:23.913149 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:23.914707 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.915232 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.915239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.915242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.915331 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.915698 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.915701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.915703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.915932 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:23.916003 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:23.916377 (+ 374us) server_negotiation.cc:299] Negotiation successful
0619 14:47:23.918295 (+ 1918us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:47:23.919241 22919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:47:23.921236 22769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6981a7854d482589ec0107fd23a7d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } peers { permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } } peers { permanent_uuid: "9953595f091d4793884be979536b31e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } }
I20260619 14:47:23.921530 22769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c6981a7854d482589ec0107fd23a7d9, State: Initialized, Role: FOLLOWER
I20260619 14:47:23.921896 22769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6981a7854d482589ec0107fd23a7d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } peers { permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } } peers { permanent_uuid: "9953595f091d4793884be979536b31e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } }
I20260619 14:47:23.922983 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.906770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33737 (local address 127.0.0.1:33522)
0619 14:47:23.907018 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.907035 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:23.907124 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:23.911785 (+ 4661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:23.911788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:23.912032 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:23.912142 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.912150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.913380 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.913384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.914792 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.914804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.915570 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.915574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.915658 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.916341 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:23.916359 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:23.917924 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:23.921396 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:23.921399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:23.921402 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:23.921649 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:23.921967 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:23.921970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:23.921972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:23.922053 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:23.922488 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:23.922491 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:23.922583 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 14:47:23.922670 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:47:23.922793 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"spinlock_wait_cycles":3456}
I20260619 14:47:23.923388 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.911624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37594 (local address 127.0.98.62:40633)
0619 14:47:23.911769 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:23.911774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:23.911786 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:47:23.911818 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:23.911822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:23.911872 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:23.911950 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:23.913049 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.913786 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.915066 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.915381 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.915544 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.918744 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:23.918748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:23.918751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:23.918777 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:23.920598 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.921115 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.921119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.921120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.921218 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.921696 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.921700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.921701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.921908 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:23.922238 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:23.922871 (+ 633us) server_negotiation.cc:299] Negotiation successful
0619 14:47:23.922990 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:47:23.923666 22769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:23.924228 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.907004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.0.98.60:33737)
0619 14:47:23.907188 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:23.907192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:23.907204 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:47:23.907242 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:23.907244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:23.907278 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:23.912269 (+ 4991us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:23.912383 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.913255 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.914984 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.915381 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.918697 (+ 3316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.919266 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:23.919274 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:23.919278 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:23.919307 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:23.921282 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.921767 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.921770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.921772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.921845 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.922186 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.922189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.922191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.922384 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:23.923350 (+ 966us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:23.923414 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 14:47:23.923547 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260619 14:47:23.927060 22926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6981a7854d482589ec0107fd23a7d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } peers { permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } } peers { permanent_uuid: "9953595f091d4793884be979536b31e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } } }
I20260619 14:47:23.927184 22926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:47:23.927615 22844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd: Bootstrap starting.
W20260619 14:47:23.929422 22937 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:47:23.929508 22937 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 19 14:47:23 dist-test-slave-cr22 krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 14:47:23 dist-test-slave-cr22 krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:47:23.934386 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.919722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33737 (local address 127.0.0.1:33528)
0619 14:47:23.919932 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.919946 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:23.920132 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:23.921073 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:23.921075 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:23.921289 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:23.921389 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.921395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.922446 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.922450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.923204 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.923216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.923839 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.923843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.923954 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.924839 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:23.924854 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:23.930489 (+ 5635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:23.932638 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:23.932642 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:23.932644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:23.932892 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:23.933283 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:23.933288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:23.933292 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:23.933386 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:23.933824 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:23.933830 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:23.933924 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 14:47:23.934019 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 14:47:23.934180 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":20}
I20260619 14:47:23.935313 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.919817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33528 (local address 127.0.98.60:33737)
0619 14:47:23.920795 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:23.920799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:23.920811 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:47:23.920843 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:23.920846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:23.920889 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:23.920965 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:23.921505 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.922268 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.923326 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.923732 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.923834 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.930661 (+ 6827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:23.930667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:23.930670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:23.930701 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:23.932469 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.933006 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.933010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.933012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.933177 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.933503 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.933506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.933508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.933726 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:23.934944 (+ 1218us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:23.935023 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 14:47:23.935140 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":893}
I20260619 14:47:23.934007 22701 master_runner.cc:186] Error getting master registration for 127.0.98.61:40755: OK, instance_id { permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" instance_seqno: 1781880443646809 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:47:23.937597 22701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:47:23.937821 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.925078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.0.98.61:40755)
0619 14:47:23.925173 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:23.925177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:23.925457 (+ 280us) server_negotiation.cc:407] Connection header received
0619 14:47:23.925496 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:23.925498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:23.925540 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:23.925612 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:23.926435 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.927113 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.928478 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.928859 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.928960 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.933694 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:23.933698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:23.933701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:23.933725 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:23.935542 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.936080 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.936083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.936084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.936176 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.936460 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.936463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.936464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.936683 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:23.936849 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:23.937541 (+ 692us) server_negotiation.cc:299] Negotiation successful
0619 14:47:23.937656 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:47:23.938012 22844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:23.938848 22844 log.cc:824] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:23.940492 22919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0: Bootstrap starting.
I20260619 14:47:23.941432 22844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd: No bootstrap required, opened a new log
I20260619 14:47:23.943274 22919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:23.944175 22919 log.cc:824] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:23.945804 22844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6981a7854d482589ec0107fd23a7d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } peers { permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } } peers { permanent_uuid: "9953595f091d4793884be979536b31e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } }
I20260619 14:47:23.946069 22844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ae621bccf6415b8eebf59e90db0cdd, State: Initialized, Role: FOLLOWER
I20260619 14:47:23.946213 22919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0: No bootstrap required, opened a new log
I20260619 14:47:23.946470 22844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6981a7854d482589ec0107fd23a7d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } peers { permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } } peers { permanent_uuid: "9953595f091d4793884be979536b31e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } }
I20260619 14:47:23.946946 22941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6981a7854d482589ec0107fd23a7d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } peers { permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } } peers { permanent_uuid: "9953595f091d4793884be979536b31e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } } }
I20260619 14:47:23.947134 22941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:47:23.947979 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:47:23.948042 22844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:23.949633 22743 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2c6981a7854d482589ec0107fd23a7d9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37594
I20260619 14:47:23.949905 22743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 14:47:23.950034 22743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:23.950103 22919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6981a7854d482589ec0107fd23a7d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } peers { permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } } peers { permanent_uuid: "9953595f091d4793884be979536b31e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } }
I20260619 14:47:23.950366 22919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9953595f091d4793884be979536b31e0, State: Initialized, Role: FOLLOWER
I20260619 14:47:23.950713 22919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6981a7854d482589ec0107fd23a7d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } peers { permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } } peers { permanent_uuid: "9953595f091d4793884be979536b31e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } }
I20260619 14:47:23.951252 22953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6981a7854d482589ec0107fd23a7d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } peers { permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } } peers { permanent_uuid: "9953595f091d4793884be979536b31e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } } }
I20260619 14:47:23.951503 22743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6981a7854d482589ec0107fd23a7d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } peers { permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } } peers { permanent_uuid: "9953595f091d4793884be979536b31e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } }
I20260619 14:47:23.951503 22953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:47:23.952090 22919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:47:23.953783 22952 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:47:23.953892 22952 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:47:23.953356 22818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c6981a7854d482589ec0107fd23a7d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "19ae621bccf6415b8eebf59e90db0cdd"
I20260619 14:47:23.954188 22818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:23.955767 22818 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c6981a7854d482589ec0107fd23a7d9 in term 1.
I20260619 14:47:23.956223 22718 leader_election.cc:302] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19ae621bccf6415b8eebf59e90db0cdd, 2c6981a7854d482589ec0107fd23a7d9; no voters:
I20260619 14:47:23.956583 22926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:23.956797 22926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [term 1 LEADER]: Becoming Leader. State: Replica: 2c6981a7854d482589ec0107fd23a7d9, State: Running, Role: LEADER
I20260619 14:47:23.957146 22926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6981a7854d482589ec0107fd23a7d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } peers { permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } } peers { permanent_uuid: "9953595f091d4793884be979536b31e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } }
I20260619 14:47:23.957218 22743 leader_election.cc:288] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [CANDIDATE]: Term 1 election: Requested vote from peers 19ae621bccf6415b8eebf59e90db0cdd (127.0.98.61:40755), 9953595f091d4793884be979536b31e0 (127.0.98.60:33737)
I20260619 14:47:23.957751 22893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c6981a7854d482589ec0107fd23a7d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9953595f091d4793884be979536b31e0"
I20260619 14:47:23.958029 22893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0 [term 0 FOLLOWER]: Advancing to term 1
W20260619 14:47:23.958717 22965 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:47:23.958802 22965 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:47:23.958917 22926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c6981a7854d482589ec0107fd23a7d9. Latest consensus state: current_term: 1 leader_uuid: "2c6981a7854d482589ec0107fd23a7d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6981a7854d482589ec0107fd23a7d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } peers { permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } } peers { permanent_uuid: "9953595f091d4793884be979536b31e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } } }
I20260619 14:47:23.959049 22926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:23.965073 22969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:23.965065 22893 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c6981a7854d482589ec0107fd23a7d9 in term 1.
I20260619 14:47:23.966217 22969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:23.967670 22850 master_runner.cc:263] No leader master found from master 9953595f091d4793884be979536b31e0
I20260619 14:47:23.967753 22850 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:47:23.968645 22776 master_runner.cc:263] No leader master found from master 19ae621bccf6415b8eebf59e90db0cdd
I20260619 14:47:23.968729 22776 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:47:23.987998 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.961838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33530 (local address 127.0.98.60:33737)
0619 14:47:23.961974 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:23.961978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:23.964192 (+ 2214us) server_negotiation.cc:407] Connection header received
0619 14:47:23.965007 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:23.965010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:23.965055 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:23.965108 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:23.968892 (+ 3784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.969579 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.973141 (+ 3562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.973498 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.973581 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.982261 (+ 8680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:23.982265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:23.982268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:23.982299 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:23.983989 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.985759 (+ 1770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.985763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.985765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.985847 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.986309 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.986313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.986315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.986554 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:23.986680 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:23.987681 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0619 14:47:23.987814 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:47:23.989971 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.962296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37604 (local address 127.0.98.62:40633)
0619 14:47:23.962497 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:23.962502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:23.963856 (+ 1354us) server_negotiation.cc:407] Connection header received
0619 14:47:23.964270 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:23.964273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:23.964326 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:23.964380 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:23.966760 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.967519 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.971975 (+ 4456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.972323 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.972405 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.979942 (+ 7537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:23.979946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:23.979949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:23.979995 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:23.981712 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.985525 (+ 3813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.985529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.985532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.985628 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.986995 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.986998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.987000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.987217 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:23.987321 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:23.989669 (+ 2348us) server_negotiation.cc:299] Negotiation successful
0619 14:47:23.989801 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260619 14:47:23.995388 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.963023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51000 (local address 127.0.98.61:40755)
0619 14:47:23.963135 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:23.963139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:23.968821 (+ 5682us) server_negotiation.cc:407] Connection header received
0619 14:47:23.970832 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:23.970836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:23.970888 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:23.970943 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:23.974357 (+ 3414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.975148 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.978828 (+ 3680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.979188 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.979269 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.991381 (+ 12112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:23.991385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:23.991388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:23.991422 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:23.993238 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.993923 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.993927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.993929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.994031 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:23.994407 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:23.994410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:23.994412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:23.994634 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:23.994741 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:23.995082 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 14:47:23.995224 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:47:24.001354 22818 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd [term 1 FOLLOWER]: Refusing update from remote peer 2c6981a7854d482589ec0107fd23a7d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:47:24.002342 22893 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0 [term 1 FOLLOWER]: Refusing update from remote peer 2c6981a7854d482589ec0107fd23a7d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:47:24.002940 22926 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:47:24.003383 22926 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9953595f091d4793884be979536b31e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:47:24.007251 22953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c6981a7854d482589ec0107fd23a7d9. Latest consensus state: current_term: 1 leader_uuid: "2c6981a7854d482589ec0107fd23a7d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6981a7854d482589ec0107fd23a7d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } peers { permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } } peers { permanent_uuid: "9953595f091d4793884be979536b31e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } } }
I20260619 14:47:24.007383 22953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:47:24.007802 22941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c6981a7854d482589ec0107fd23a7d9. Latest consensus state: current_term: 1 leader_uuid: "2c6981a7854d482589ec0107fd23a7d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6981a7854d482589ec0107fd23a7d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } peers { permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } } peers { permanent_uuid: "9953595f091d4793884be979536b31e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } } }
I20260619 14:47:24.007947 22941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:47:24.021699 22966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c6981a7854d482589ec0107fd23a7d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6981a7854d482589ec0107fd23a7d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } peers { permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } } peers { permanent_uuid: "9953595f091d4793884be979536b31e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } } }
I20260619 14:47:24.021881 22966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:24.023191 22953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c6981a7854d482589ec0107fd23a7d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6981a7854d482589ec0107fd23a7d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } peers { permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } } peers { permanent_uuid: "9953595f091d4793884be979536b31e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } } }
I20260619 14:47:24.023317 22953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:47:24.023765 22977 mvcc.cc:205] Tried to move back new op lower bound from 7298582298619502592 to 7298582298451881984. Current Snapshot: MvccSnapshot[applied={T|T < 7298582298619502592}]
I20260619 14:47:24.023921 22978 mvcc.cc:205] Tried to move back new op lower bound from 7298582298619502592 to 7298582298451881984. Current Snapshot: MvccSnapshot[applied={T|T < 7298582298619502592}]
I20260619 14:47:24.025038 22926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c6981a7854d482589ec0107fd23a7d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6981a7854d482589ec0107fd23a7d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } peers { permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } } peers { permanent_uuid: "9953595f091d4793884be979536b31e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } } }
I20260619 14:47:24.025163 22926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:24.026345 22941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c6981a7854d482589ec0107fd23a7d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6981a7854d482589ec0107fd23a7d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40633 } } peers { permanent_uuid: "19ae621bccf6415b8eebf59e90db0cdd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40755 } } peers { permanent_uuid: "9953595f091d4793884be979536b31e0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33737 } } }
I20260619 14:47:24.026470 22941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:47:24.027155 22979 mvcc.cc:205] Tried to move back new op lower bound from 7298582298619502592 to 7298582298451881984. Current Snapshot: MvccSnapshot[applied={T|T < 7298582298619502592}]
I20260619 14:47:24.027890 22969 catalog_manager.cc:1347] Generated new cluster ID: 513398cc49874ee0bc9cd299772bbf08
I20260619 14:47:24.027962 22969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:24.086547 22969 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:24.087916 22969 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:24.106948 22969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c6981a7854d482589ec0107fd23a7d9: Generated new TSK 0
I20260619 14:47:24.107919 22969 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:47:24.148387 22772 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:24.145464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37616 (local address 127.0.98.62:40633)
0619 14:47:24.145591 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:24.145595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:24.147284 (+ 1689us) server_negotiation.cc:407] Connection header received
0619 14:47:24.147525 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:24.147529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:24.147582 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:24.147635 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:24.148249 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37616: BlockingRecv error: recv got EOF from 127.0.0.1:37616 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 14:47:24.149881 22848 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:24.146218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.0.98.61:40755)
0619 14:47:24.146360 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:24.146364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:24.149015 (+ 2651us) server_negotiation.cc:407] Connection header received
0619 14:47:24.149171 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:24.149174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:24.149222 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:24.149272 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:24.149729 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51006: BlockingRecv error: recv got EOF from 127.0.0.1:51006 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 14:47:24.151324 22922 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:24.146764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33546 (local address 127.0.98.60:33737)
0619 14:47:24.146896 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:24.146901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:24.150487 (+ 3586us) server_negotiation.cc:407] Connection header received
0619 14:47:24.150647 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:24.150651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:24.150696 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:24.150745 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:24.151206 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33546: BlockingRecv error: recv got EOF from 127.0.0.1:33546 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 14:47:24.155964 22848 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:24.154391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51012 (local address 127.0.98.61:40755)
0619 14:47:24.154487 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:24.154491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:24.154966 (+ 475us) server_negotiation.cc:407] Connection header received
0619 14:47:24.155205 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:24.155209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:24.155280 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:24.155328 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:24.155844 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51012: BlockingRecv error: recv got EOF from 127.0.0.1:51012 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 14:47:24.160413 22848 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:24.159571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.0.98.61:40755)
0619 14:47:24.159681 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:24.159684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:24.159694 (+ 10us) server_negotiation.cc:407] Connection header received
0619 14:47:24.159728 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:24.159730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:24.159765 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:24.159860 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:24.160304 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51022: BlockingRecv error: recv got EOF from 127.0.0.1:51022 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 14:47:24.165406 22848 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:24.164122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51026 (local address 127.0.98.61:40755)
0619 14:47:24.164222 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:24.164225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:24.164434 (+ 209us) server_negotiation.cc:407] Connection header received
0619 14:47:24.164593 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:24.164601 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:24.164640 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:24.164728 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:24.165306 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51026: BlockingRecv error: recv got EOF from 127.0.0.1:51026 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 14:47:24.172088 22848 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:24.170909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51028 (local address 127.0.98.61:40755)
0619 14:47:24.171047 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:24.171052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:24.171265 (+ 213us) server_negotiation.cc:407] Connection header received
0619 14:47:24.171416 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:24.171419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:24.171503 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:24.171584 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:24.171984 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51028: BlockingRecv error: recv got EOF from 127.0.0.1:51028 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 14:47:24.183198 22848 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:24.181967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51032 (local address 127.0.98.61:40755)
0619 14:47:24.182115 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:24.182120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:24.182351 (+ 231us) server_negotiation.cc:407] Connection header received
0619 14:47:24.182521 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:24.182524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:24.182576 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:24.182664 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:24.183085 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51032: BlockingRecv error: recv got EOF from 127.0.0.1:51032 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 14:47:24.202478 22848 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:24.201167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51034 (local address 127.0.98.61:40755)
0619 14:47:24.201317 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:24.201321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:24.201424 (+ 103us) server_negotiation.cc:407] Connection header received
0619 14:47:24.201540 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:24.201544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:24.201606 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:24.201717 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:24.202343 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51034: BlockingRecv error: recv got EOF from 127.0.0.1:51034 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 14:47:24.238056 22848 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:24.236796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51042 (local address 127.0.98.61:40755)
0619 14:47:24.236945 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:24.236950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:24.237189 (+ 239us) server_negotiation.cc:407] Connection header received
0619 14:47:24.237288 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:24.237291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:24.237348 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:24.237442 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:24.237947 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51042: BlockingRecv error: recv got EOF from 127.0.0.1:51042 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 14:47:24.305758 22848 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:24.304567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51056 (local address 127.0.98.61:40755)
0619 14:47:24.304754 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:24.304758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:24.304883 (+ 125us) server_negotiation.cc:407] Connection header received
0619 14:47:24.305017 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:24.305019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:24.305070 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:24.305163 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:24.305652 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51056: BlockingRecv error: recv got EOF from 127.0.0.1:51056 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260619 14:47:24.437258 22848 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:24.436127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51062 (local address 127.0.98.61:40755)
0619 14:47:24.436285 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:24.436289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:24.436410 (+ 121us) server_negotiation.cc:407] Connection header received
0619 14:47:24.436548 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:24.436551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:24.436602 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:24.436691 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:24.437152 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51062: BlockingRecv error: recv got EOF from 127.0.0.1:51062 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 14:47:24.697235 22848 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:24.695954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.0.98.61:40755)
0619 14:47:24.696191 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:24.696195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:24.696213 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:47:24.696393 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:24.696397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:24.696480 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:24.696578 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:24.697118 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51066: BlockingRecv error: recv got EOF from 127.0.0.1:51066 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:47:24.955407 22952 catalog_manager.cc:1259] Loaded cluster ID: 513398cc49874ee0bc9cd299772bbf08
I20260619 14:47:24.955523 22952 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd: loading cluster ID for follower catalog manager: success
I20260619 14:47:24.957284 22952 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd: acquiring CA information for follower catalog manager: success
I20260619 14:47:24.958474 22952 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 19ae621bccf6415b8eebf59e90db0cdd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:47:24.959992 22965 catalog_manager.cc:1259] Loaded cluster ID: 513398cc49874ee0bc9cd299772bbf08
I20260619 14:47:24.960054 22965 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0: loading cluster ID for follower catalog manager: success
I20260619 14:47:24.961602 22965 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0: acquiring CA information for follower catalog manager: success
I20260619 14:47:24.962635 22965 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9953595f091d4793884be979536b31e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:47:25.215603 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:25.211479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51072 (local address 127.0.98.61:40755)
0619 14:47:25.211739 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:25.211743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:25.211758 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:47:25.211866 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:25.211868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:25.212050 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:25.212154 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:47:25.212463 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.212998 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.213925 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.214189 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.214278 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:25.214559 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:47:25.214997 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:47:25.215158 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:25.215294 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 14:47:25.215382 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":117,"threads_started":1}
I20260619 14:47:25.222748 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:25.218981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.0.98.60:33737)
0619 14:47:25.219232 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:25.219235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:25.219248 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:47:25.219321 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:25.219324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:25.219456 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:25.219550 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:47:25.219771 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.220278 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.221141 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.221387 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.221465 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:25.221728 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:47:25.222194 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:47:25.222285 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:25.222454 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 14:47:25.222506 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":105,"threads_started":1}
I20260619 14:47:25.230564 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:25.226327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33568 (local address 127.0.98.60:33737)
0619 14:47:25.226456 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:25.226460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:25.226849 (+ 389us) server_negotiation.cc:407] Connection header received
0619 14:47:25.227036 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:25.227040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:25.227091 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:25.227163 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:47:25.227432 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.228140 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.229208 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.229506 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.229620 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:25.229876 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:47:25.229959 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:47:25.230028 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:25.230366 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 14:47:25.230419 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:47:25.230710 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:25.226498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51074 (local address 127.0.98.61:40755)
0619 14:47:25.226616 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:25.226621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:25.226927 (+ 306us) server_negotiation.cc:407] Connection header received
0619 14:47:25.227069 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:25.227072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:25.227122 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:25.227212 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:47:25.227632 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.228330 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.229375 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.229730 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.229826 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:25.230166 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:47:25.230242 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:47:25.230313 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:25.230486 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 14:47:25.230557 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:47:25.231627 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:25.226212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37630 (local address 127.0.98.62:40633)
0619 14:47:25.227178 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:25.227182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:25.227196 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:47:25.227285 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:25.227288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:25.227471 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:25.227645 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:47:25.227999 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.228812 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.229939 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.230293 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.230391 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:25.230723 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:47:25.231211 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:47:25.231307 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:25.231402 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 14:47:25.231483 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":70,"threads_started":1}
I20260619 14:47:25.234876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 22701
I20260619 14:47:25.241448 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 22776
I20260619 14:47:25.247682 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 22850
2026-06-19T14:47:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3253 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:25 dist-test-slave-cr22 krb5kdc[23066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:25 dist-test-slave-cr22 krb5kdc[23066](info): set up 2 sockets
Jun 19 14:47:25 dist-test-slave-cr22 krb5kdc[23066](info): commencing operation
krb5kdc: starting...
W20260619 14:47:26.315912 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:26 dist-test-slave-cr22 krb5kdc[23066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:47:26.443039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44573
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44573,127.0.98.61:43113,127.0.98.60:40061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:26.559787 23079 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:47:26.560127 23079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:26.560151 23079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:26.560191 23079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:26.564328 23079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:26.564392 23079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:26.564411 23079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:26.564428 23079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:26.564443 23079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:26.569797 23079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44573,127.0.98.61:43113,127.0.98.60:40061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:26.570845 23079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:26.571985 23079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:47:26.580857 23079 server_base.cc:1034] running on GCE node
W20260619 14:47:26.580857 23084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:26.580857 23085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:26.581040 23087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:26.581686 23079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:26.582789 23079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:26.583989 23079 hybrid_clock.cc:648] HybridClock initialized: now 1781880446583894 us; error 101 us; skew 500 ppm
Jun 19 14:47:26 dist-test-slave-cr22 krb5kdc[23066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:26.586874 23079 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:47:26.588117 23079 webserver.cc:460] Webserver started at http://127.0.98.62:45429/ using document root <none> and password file <none>
I20260619 14:47:26.588650 23079 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:26.588711 23079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:26.588892 23079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:26.590965 23079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-0/data/instance:
uuid: "ebb6c7cd1cf840308e526575034f9993"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-cr22"
server_key: "843c1ee19a96b82128085a0f9a20843a"
server_key_iv: "30c20642aad134882d3b6832fd88fcf4"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.591741 23079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-0/wal/instance:
uuid: "ebb6c7cd1cf840308e526575034f9993"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-cr22"
server_key: "843c1ee19a96b82128085a0f9a20843a"
server_key_iv: "30c20642aad134882d3b6832fd88fcf4"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.595858 23079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:47:26.598551 23094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:26.599995 23079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:26.600229 23079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-0/wal
uuid: "ebb6c7cd1cf840308e526575034f9993"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-cr22"
server_key: "843c1ee19a96b82128085a0f9a20843a"
server_key_iv: "30c20642aad134882d3b6832fd88fcf4"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.600358 23079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:26.621374 23079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:26.624900 23079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:26.625139 23079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:26.632807 23079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44573
I20260619 14:47:26.633103 23146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44573 every 8 connection(s)
I20260619 14:47:26.633901 23079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:26.636762 23147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:26.640697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 23079
I20260619 14:47:26.640820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:26.640594 23147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:47:26.641116 392 external_mini_cluster.cc:1428] Setting key ae1634cbb0bc920b02227025b00aae10
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 14:47:26 dist-test-slave-cr22 krb5kdc[23066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:47:26.660672 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.643334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44573 (local address 127.0.0.1:57026)
0619 14:47:26.643980 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.644040 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:26.644810 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:26.645506 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:26.645514 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:26.646115 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:26.646503 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:26.646528 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.648302 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.648305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:26.649107 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:26.649116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.650017 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.650022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:26.650159 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:26.650788 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:26.650804 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:26.655281 (+ 4477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:26.657401 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:26.657414 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:26.657428 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:26.657753 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:26.658151 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:26.658158 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:26.658162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:26.658334 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:26.659021 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:26.659040 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:26.659230 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 14:47:26.659471 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 14:47:26.659894 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":71,"spinlock_wait_cycles":8192,"thread_start_us":230,"threads_started":1}
I20260619 14:47:26.660671 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.643335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57026 (local address 127.0.98.62:44573)
0619 14:47:26.643799 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:26.643820 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:26.644149 (+ 329us) server_negotiation.cc:407] Connection header received
0619 14:47:26.645119 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:26.645123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:26.645304 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:26.645454 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:26.646706 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.648143 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.649235 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.649842 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.650017 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:26.655548 (+ 5531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:26.655554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:26.655565 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:26.655587 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:26.657286 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:26.657862 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:26.657867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:26.657868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:26.658044 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:26.658451 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:26.658454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:26.658455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:26.658840 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:26.659019 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:26.659573 (+ 554us) server_negotiation.cc:299] Negotiation successful
0619 14:47:26.659784 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":330,"spinlock_wait_cycles":7040,"thread_start_us":168,"threads_started":1}
I20260619 14:47:26.663189 23147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:47:26.663894 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.663662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43113 (local address 127.0.0.1:47984)
0619 14:47:26.663773 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.663839 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 14:47:26.664075 23095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43113: connect: Connection refused (error 111)
I20260619 14:47:26.664994 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.664720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43113 (local address 127.0.0.1:47990)
0619 14:47:26.664860 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.664919 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 14:47:26.665287 23147 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43113: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43113: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 14:47:26.671617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43113
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44573,127.0.98.61:43113,127.0.98.60:40061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:47:26.714597 23147 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } attempt: 1
I20260619 14:47:26.715657 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.715209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43113 (local address 127.0.0.1:48006)
0619 14:47:26.715408 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.715540 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 14:47:26.716602 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.716298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43113 (local address 127.0.0.1:48018)
0619 14:47:26.716449 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.716516 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 14:47:26.716773 23147 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43113: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43113: connect: Connection refused (error 111)
I20260619 14:47:26.784958 23147 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } attempt: 2
I20260619 14:47:26.785853 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.785483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43113 (local address 127.0.0.1:48034)
0619 14:47:26.785694 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.785764 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 14:47:26.786777 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.786556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43113 (local address 127.0.0.1:48036)
0619 14:47:26.786669 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.786717 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 14:47:26.786978 23147 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43113: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43113: connect: Connection refused (error 111)
W20260619 14:47:26.790030 23154 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:47:26.790289 23154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:26.790340 23154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:26.790406 23154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:26.794479 23154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:26.794538 23154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:26.794557 23154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:26.794574 23154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:26.794620 23154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:26.800184 23154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44573,127.0.98.61:43113,127.0.98.60:40061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:26.801255 23154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:26.802289 23154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:26.811138 23160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:26.811177 23162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:26.811229 23154 server_base.cc:1034] running on GCE node
W20260619 14:47:26.811177 23159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:26.812041 23154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:26.813112 23154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:26.814311 23154 hybrid_clock.cc:648] HybridClock initialized: now 1781880446814291 us; error 33 us; skew 500 ppm
Jun 19 14:47:26 dist-test-slave-cr22 krb5kdc[23066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:26.817131 23154 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 14:47:26.818002 23154 webserver.cc:460] Webserver started at http://127.0.98.61:34485/ using document root <none> and password file <none>
I20260619 14:47:26.818496 23154 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:26.818552 23154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:26.818708 23154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:26.820731 23154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-1/data/instance:
uuid: "368fb0a65263421dbec5c3498b152093"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-cr22"
server_key: "335ddc5b1e22bf465443a0f235076c2b"
server_key_iv: "ee977efaa765cddbf8ff7c6b8fc52dc0"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.821240 23154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-1/wal/instance:
uuid: "368fb0a65263421dbec5c3498b152093"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-cr22"
server_key: "335ddc5b1e22bf465443a0f235076c2b"
server_key_iv: "ee977efaa765cddbf8ff7c6b8fc52dc0"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.825217 23154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:47:26.827850 23169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:26.829161 23154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:26.829322 23154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-1/wal
uuid: "368fb0a65263421dbec5c3498b152093"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-cr22"
server_key: "335ddc5b1e22bf465443a0f235076c2b"
server_key_iv: "ee977efaa765cddbf8ff7c6b8fc52dc0"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.829449 23154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:26.850135 23154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:26.853566 23154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:26.853820 23154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:26.861903 23154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43113
I20260619 14:47:26.862375 23221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43113 every 8 connection(s)
I20260619 14:47:26.863013 23154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-1/data/info.pb
I20260619 14:47:26.865990 23222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:26.868767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 23154
I20260619 14:47:26.868942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-1/wal/instance
I20260619 14:47:26.869307 392 external_mini_cluster.cc:1428] Setting key 1977f6713408956c7e698ad81f2d4601
I20260619 14:47:26.870287 23222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:47:26.878180 23147 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } attempt: 3
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 14:47:26 dist-test-slave-cr22 krb5kdc[23066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 14:47:26 dist-test-slave-cr22 krb5kdc[23066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:47:26.889096 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.872787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.0.98.62:44573)
0619 14:47:26.872959 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:26.872964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:26.874017 (+ 1053us) server_negotiation.cc:407] Connection header received
0619 14:47:26.875766 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:26.875769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:26.875819 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:26.875918 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:26.877775 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.878382 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.879985 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.880249 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.880371 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:26.883468 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:26.883473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:26.883478 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:26.883511 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:26.885587 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:26.886323 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:26.886330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:26.886335 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:26.886436 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:26.886833 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:26.886837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:26.886839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:26.887038 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:26.887149 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:26.888607 (+ 1458us) server_negotiation.cc:299] Negotiation successful
0619 14:47:26.888777 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:47:26.889259 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.873200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44573 (local address 127.0.0.1:57036)
0619 14:47:26.873824 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.873889 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:26.875396 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:26.876154 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:26.876164 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:26.876788 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:26.877573 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:26.877595 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.878553 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.878556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:26.879863 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:26.879871 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.880409 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.880413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:26.880592 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:26.881257 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:26.881274 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:26.883283 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:26.885721 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:26.885728 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:26.885741 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:26.886113 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:26.886552 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:26.886557 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:26.886562 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:26.886738 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:26.887149 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:26.887157 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:26.887549 (+ 392us) client_negotiation.cc:769] Sending connection context
0619 14:47:26.887775 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 14:47:26.888142 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":163,"threads_started":1}
I20260619 14:47:26.889884 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.878681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43113 (local address 127.0.0.1:48042)
0619 14:47:26.878881 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.878899 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:26.878992 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:26.879681 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:26.879683 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:26.879834 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:26.879934 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:26.879939 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.881502 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.881506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:26.882409 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:26.882422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.883191 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.883194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:26.883311 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:26.883884 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:26.883896 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:26.885458 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:26.888147 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:26.888150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:26.888152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:26.888403 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:26.888733 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:26.888736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:26.888738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:26.888821 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:26.889353 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:26.889357 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:26.889486 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 14:47:26.889589 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 14:47:26.889722 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260619 14:47:26.890530 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.878788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48042 (local address 127.0.98.61:43113)
0619 14:47:26.879197 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:26.879207 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:26.879231 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:47:26.879271 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:26.879276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:26.879499 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:26.879663 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:26.880076 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.881294 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.882575 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.883053 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.883184 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:26.885626 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:26.885633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:26.885655 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:26.885701 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:26.888027 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:26.888547 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:26.888553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:26.888555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:26.888635 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:26.888916 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:26.888919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:26.888921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:26.889179 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:26.889300 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:26.890129 (+ 829us) server_negotiation.cc:299] Negotiation successful
0619 14:47:26.890311 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":190,"threads_started":1}
I20260619 14:47:26.890956 23222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:47:26.892704 23147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:47:26.893373 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.893148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40061 (local address 127.0.0.1:58428)
0619 14:47:26.893255 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.893306 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 14:47:26.894089 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.893830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40061 (local address 127.0.0.1:58440)
0619 14:47:26.893978 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.894031 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260619 14:47:26.894246 23147 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40061: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40061: connect: Connection refused (error 111)
Jun 19 14:47:26 dist-test-slave-cr22 krb5kdc[23066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 14:47:26.904448 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
/tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40061
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44573,127.0.98.61:43113,127.0.98.60:40061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:47:26.904867 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.891731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48044 (local address 127.0.98.61:43113)
0619 14:47:26.891863 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:26.891868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:26.891882 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:47:26.891917 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:26.891920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:26.891967 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:26.892087 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:26.892775 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.893635 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.894637 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.894904 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.894991 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:26.901118 (+ 6127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:26.901122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:26.901128 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:26.901151 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:26.902747 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:26.903324 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:26.903329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:26.903331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:26.903421 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:26.903857 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:26.903860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:26.903862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:26.904119 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:26.904235 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:26.904528 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 14:47:26.904744 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":46}
I20260619 14:47:26.904867 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.891500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43113 (local address 127.0.0.1:48044)
0619 14:47:26.891694 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.891709 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:26.891782 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:26.892112 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:26.892115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:26.892282 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:26.892390 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:26.892396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.893759 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.893763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:26.894530 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:26.894539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.895011 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.895014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:26.895100 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:26.899322 (+ 4222us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:26.899335 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:26.900817 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:26.902883 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:26.902885 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:26.902887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:26.903161 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:26.903615 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:26.903621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:26.903626 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:26.903725 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:26.904225 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:26.904228 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:26.904322 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 14:47:26.904448 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 14:47:26.904678 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260619 14:47:26.906064 23222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:47:26.906828 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.906535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40061 (local address 127.0.0.1:58446)
0619 14:47:26.906683 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.906770 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 14:47:26.907043 23170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40061: connect: Connection refused (error 111)
I20260619 14:47:26.907897 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.907686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40061 (local address 127.0.0.1:58448)
0619 14:47:26.907799 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.907847 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 14:47:26.908066 23222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40061: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40061: connect: Connection refused (error 111)
I20260619 14:47:26.925464 23147 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } attempt: 1
I20260619 14:47:26.926451 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.926139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40061 (local address 127.0.0.1:58458)
0619 14:47:26.926305 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.926370 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 14:47:26.927219 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.926973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40061 (local address 127.0.0.1:58470)
0619 14:47:26.927125 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.927168 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260619 14:47:26.927361 23147 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40061: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40061: connect: Connection refused (error 111)
I20260619 14:47:26.957325 23222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } attempt: 1
I20260619 14:47:26.958365 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.958031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40061 (local address 127.0.0.1:58474)
0619 14:47:26.958222 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.958284 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 14:47:26.959218 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.958926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40061 (local address 127.0.0.1:58486)
0619 14:47:26.959111 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.959165 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260619 14:47:26.959362 23222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40061: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40061: connect: Connection refused (error 111)
I20260619 14:47:27.002545 23147 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } attempt: 2
I20260619 14:47:27.003574 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.003141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40061 (local address 127.0.0.1:58492)
0619 14:47:27.003306 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:27.003423 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 14:47:27.004375 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.004130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40061 (local address 127.0.0.1:58508)
0619 14:47:27.004243 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:27.004299 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 14:47:27.004591 23147 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40061: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40061: connect: Connection refused (error 111)
W20260619 14:47:27.018370 23228 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:47:27.018658 23228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:27.018683 23228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:27.018778 23228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:27.022953 23228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:27.023250 23228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:27.023279 23228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:27.023365 23228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:27.023401 23228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:27.027536 23222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } attempt: 2
I20260619 14:47:27.028405 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.028098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40061 (local address 127.0.0.1:58524)
0619 14:47:27.028257 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:27.028322 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 14:47:27.029229 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.028972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40061 (local address 127.0.0.1:58540)
0619 14:47:27.029132 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:27.029178 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 14:47:27.029379 23222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40061: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40061: connect: Connection refused (error 111)
I20260619 14:47:27.030081 23228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44573,127.0.98.61:43113,127.0.98.60:40061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:47:27.031347 23228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:27.032481 23228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:27.041595 23234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:27.041661 23235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:27.041692 23237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:27.041846 23228 server_base.cc:1034] running on GCE node
I20260619 14:47:27.042560 23228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:27.043768 23228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:27.044950 23228 hybrid_clock.cc:648] HybridClock initialized: now 1781880447044930 us; error 35 us; skew 500 ppm
Jun 19 14:47:27 dist-test-slave-cr22 krb5kdc[23066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:27.047842 23228 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 14:47:27.048892 23228 webserver.cc:460] Webserver started at http://127.0.98.60:41053/ using document root <none> and password file <none>
I20260619 14:47:27.049463 23228 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:27.049564 23228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:27.049777 23228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:27.052016 23228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-2/data/instance:
uuid: "7698476615364546a0c818170529903b"
format_stamp: "Formatted at 2026-06-19 14:47:27 on dist-test-slave-cr22"
server_key: "43c596812724333ebc30e2da044cd857"
server_key_iv: "3cb45672592d70afe97b0f1ce83d50bc"
server_key_version: "encryptionkey@0"
I20260619 14:47:27.052594 23228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-2/wal/instance:
uuid: "7698476615364546a0c818170529903b"
format_stamp: "Formatted at 2026-06-19 14:47:27 on dist-test-slave-cr22"
server_key: "43c596812724333ebc30e2da044cd857"
server_key_iv: "3cb45672592d70afe97b0f1ce83d50bc"
server_key_version: "encryptionkey@0"
I20260619 14:47:27.056605 23228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:47:27.059340 23244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:27.060644 23228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:47:27.060786 23228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-2/wal
uuid: "7698476615364546a0c818170529903b"
format_stamp: "Formatted at 2026-06-19 14:47:27 on dist-test-slave-cr22"
server_key: "43c596812724333ebc30e2da044cd857"
server_key_iv: "3cb45672592d70afe97b0f1ce83d50bc"
server_key_version: "encryptionkey@0"
I20260619 14:47:27.060904 23228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:27.077512 23228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:27.080567 23228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:27.080771 23228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:27.088006 23228 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40061
I20260619 14:47:27.088442 23296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40061 every 8 connection(s)
I20260619 14:47:27.089015 23228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-2/data/info.pb
I20260619 14:47:27.091925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu as pid 23228
I20260619 14:47:27.092082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ9bYsM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880155260304-392-0/minicluster-data/master-2/wal/instance
I20260619 14:47:27.092108 23297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:27.092448 392 external_mini_cluster.cc:1428] Setting key 69efbcab0d0e1914961ac8f02e66f27d
I20260619 14:47:27.098004 23297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:47:27 dist-test-slave-cr22 krb5kdc[23066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:47:27.103826 23147 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } attempt: 3
Jun 19 14:47:27 dist-test-slave-cr22 krb5kdc[23066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:47:27.110256 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.093819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58556 (local address 127.0.98.60:40061)
0619 14:47:27.094234 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.094262 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.094307 (+ 45us) server_negotiation.cc:407] Connection header received
0619 14:47:27.095269 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.095295 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.095805 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.096793 (+ 988us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.097415 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.098883 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.099944 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.100612 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.100795 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:27.103964 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:27.103971 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:27.103987 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:27.104034 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:27.106252 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.106790 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.106800 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.106811 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.107068 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.107423 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.107427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.107429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.107956 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:27.108159 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:27.108852 (+ 693us) server_negotiation.cc:299] Negotiation successful
0619 14:47:27.109187 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":105,"threads_started":1}
Jun 19 14:47:27 dist-test-slave-cr22 krb5kdc[23066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:47:27.111966 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.099954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.0.98.62:44573)
0619 14:47:27.100093 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.100098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.100364 (+ 266us) server_negotiation.cc:407] Connection header received
0619 14:47:27.100524 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.100528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.100578 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.100664 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.101514 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.102452 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.103517 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.103957 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.104072 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:27.107990 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:27.107994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:27.107997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:27.108025 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:27.109742 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.110270 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.110276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.110280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.110368 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.110674 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.110677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.110679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.110934 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:27.111054 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:27.111692 (+ 638us) server_negotiation.cc:299] Negotiation successful
0619 14:47:27.111804 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:47:27.111999 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.099856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44573 (local address 127.0.0.1:57046)
0619 14:47:27.100248 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:27.100287 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:27.100426 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:27.100716 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:27.100722 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:27.101113 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:27.101337 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:27.101347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.102611 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.102614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:27.103366 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:27.103376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.104164 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.104169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:27.104901 (+ 732us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:27.105767 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:27.105781 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:27.107804 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:27.109842 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:27.109854 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:27.109861 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:27.110157 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:27.110493 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:27.110496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:27.110498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:27.110590 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:27.111308 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:27.111314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:27.111469 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 14:47:27.111619 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 14:47:27.111806 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":147,"threads_started":1}
I20260619 14:47:27.113682 23297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:47:27.115031 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.104344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40061 (local address 127.0.0.1:58560)
0619 14:47:27.104491 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:27.104531 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:27.104616 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:27.105241 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:27.105244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:27.105542 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:27.105655 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:27.105663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.106887 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.106892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:27.107740 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:27.107755 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.108615 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.108621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:27.108726 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:27.109252 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:27.109267 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:27.110671 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:27.113351 (+ 2680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:27.113354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:27.113356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:27.113602 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:27.113919 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:27.113922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:27.113924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:27.114048 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:27.114513 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:27.114516 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:27.114617 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 14:47:27.114723 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 14:47:27.114865 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 14:47:27.115077 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.104669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58560 (local address 127.0.98.60:40061)
0619 14:47:27.104935 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.104961 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.104976 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:47:27.105012 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.105017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.105061 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.105166 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.105777 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.106761 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.107875 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.108233 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.108363 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:27.110784 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:27.110787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:27.110790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:27.110809 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:27.113236 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.113729 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.113733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.113734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.113821 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.114161 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.114166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.114168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.114399 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:27.114517 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:27.114802 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 14:47:27.114930 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":130,"threads_started":1}
I20260619 14:47:27.119127 23147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993: Bootstrap starting.
Jun 19 14:47:27 dist-test-slave-cr22 krb5kdc[23066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 14:47:27 dist-test-slave-cr22 krb5kdc[23066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:47:27.120535 23222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } attempt: 3
I20260619 14:47:27.122237 23147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:27.123260 23147 log.cc:824] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:27.123873 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.114457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48050 (local address 127.0.98.61:43113)
0619 14:47:27.114606 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.114620 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.114635 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:47:27.114787 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.114791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.114846 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.114898 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.115881 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.116411 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.117470 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.117739 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.117856 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:27.119951 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:27.119956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:27.119959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:27.120000 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:27.121783 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.122353 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.122358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.122360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.122458 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.122822 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.122827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.122831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.123051 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:27.123141 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:27.123609 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 14:47:27.123737 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:47:27.124313 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.114146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43113 (local address 127.0.0.1:48050)
0619 14:47:27.114592 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:27.114605 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:27.114665 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:27.115052 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:27.115056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:27.115613 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:27.115749 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:27.115757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.116535 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.116540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:27.117368 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:27.117380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.117859 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.117863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:27.117959 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:27.118439 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:27.118453 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:27.119784 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:27.121883 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:27.121886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:27.121888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:27.122171 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:27.122591 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:27.122594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:27.122596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:27.122681 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:27.123216 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:27.123219 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:27.123317 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 14:47:27.124039 (+ 722us) client_negotiation.cc:240] Negotiation successful
0619 14:47:27.124186 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387}
I20260619 14:47:27.124989 23297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:47:27.126250 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.113367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.0.98.62:44573)
0619 14:47:27.113469 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.113484 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.113976 (+ 492us) server_negotiation.cc:407] Connection header received
0619 14:47:27.114014 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.114016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.114054 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.114142 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.114921 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.115666 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.116860 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.117213 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.117315 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:27.120998 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:27.121003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:27.121007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:27.121040 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:27.123957 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.124915 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.124918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.124920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.125014 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.125328 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.125331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.125332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.125561 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:27.125780 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:27.126017 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 14:47:27.126152 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Jun 19 14:47:27 dist-test-slave-cr22 krb5kdc[23066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:47:27.128046 23147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993: No bootstrap required, opened a new log
I20260619 14:47:27.131552 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.120978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40061 (local address 127.0.0.1:58572)
0619 14:47:27.121357 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:27.121375 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:27.121466 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:27.122128 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:27.122131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:27.122308 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:27.122731 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:27.122740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.124189 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.124192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:27.124940 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:27.124950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.125581 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.125587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:27.125684 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:27.126132 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:27.126151 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:27.127842 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:27.129871 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:27.129874 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:27.129876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:27.130114 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:27.130439 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:27.130442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:27.130444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:27.130536 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:27.131019 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:27.131024 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:27.131114 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 14:47:27.131194 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 14:47:27.131340 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280}
Jun 19 14:47:27 dist-test-slave-cr22 krb5kdc[23066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:47:27.132472 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.121321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58572 (local address 127.0.98.60:40061)
0619 14:47:27.121928 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.121931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.121951 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:47:27.121979 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.121981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.122016 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.122103 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.122880 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.124046 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.125083 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.125452 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.125882 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:27.127978 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:27.127982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:27.127985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:27.128024 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:27.129769 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.130252 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.130255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.130257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.130341 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.130674 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.130682 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.130687 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.130903 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:27.132094 (+ 1191us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:27.132170 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 14:47:27.132276 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"spinlock_wait_cycles":1920}
I20260619 14:47:27.132630 23147 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb6c7cd1cf840308e526575034f9993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } } peers { permanent_uuid: "368fb0a65263421dbec5c3498b152093" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } } peers { permanent_uuid: "7698476615364546a0c818170529903b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } }
I20260619 14:47:27.132898 23147 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb6c7cd1cf840308e526575034f9993, State: Initialized, Role: FOLLOWER
I20260619 14:47:27.133291 23147 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb6c7cd1cf840308e526575034f9993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } } peers { permanent_uuid: "368fb0a65263421dbec5c3498b152093" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } } peers { permanent_uuid: "7698476615364546a0c818170529903b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } }
I20260619 14:47:27.134801 23147 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:27.135725 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.125477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40061 (local address 127.0.0.1:58582)
0619 14:47:27.125623 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:27.125637 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:27.125713 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:27.126990 (+ 1277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:27.126992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:27.127134 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:27.127191 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:27.127195 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.128200 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.128204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:27.129626 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:27.129638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.130328 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.130331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:27.130428 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:27.130843 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:27.130856 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:27.132276 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:27.134300 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:27.134302 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:27.134303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:27.134459 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:27.134736 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:27.134739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:27.134741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:27.134813 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:27.135187 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:27.135190 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:27.135267 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 14:47:27.135371 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 14:47:27.135545 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 14:47:27.135743 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.125516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.0.98.60:40061)
0619 14:47:27.125618 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.125622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.126735 (+ 1113us) server_negotiation.cc:407] Connection header received
0619 14:47:27.126778 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.126782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.126826 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.126899 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.127296 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.128091 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.129758 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.130139 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.130233 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:27.132656 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:27.132659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:27.132662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:27.132684 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:27.134213 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.134551 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.134556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.134558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.134641 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.134930 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.134933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.134934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.135091 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:27.135195 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:27.135426 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 14:47:27.135608 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":33}
I20260619 14:47:27.136236 23222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093: Bootstrap starting.
I20260619 14:47:27.138728 23297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b: Bootstrap starting.
I20260619 14:47:27.139160 23222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093: Neither blocks nor log segments found. Creating new log.
W20260619 14:47:27.140173 23315 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:47:27.140296 23315 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:47:27.140080 23222 log.cc:824] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:27.141301 23297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b: Neither blocks nor log segments found. Creating new log.
Jun 19 14:47:27 dist-test-slave-cr22 krb5kdc[23066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:47:27.142221 23297 log.cc:824] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:27.143366 23304 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb6c7cd1cf840308e526575034f9993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } } peers { permanent_uuid: "368fb0a65263421dbec5c3498b152093" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } } peers { permanent_uuid: "7698476615364546a0c818170529903b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } } }
I20260619 14:47:27.143567 23304 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:47:27.139964 23079 master_runner.cc:186] Error getting master registration for 127.0.98.61:43113: OK, instance_id { permanent_uuid: "368fb0a65263421dbec5c3498b152093" instance_seqno: 1781880446859545 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:47:27.143847 23079 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:47:27.145751 23222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093: No bootstrap required, opened a new log
I20260619 14:47:27.146034 23297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b: No bootstrap required, opened a new log
I20260619 14:47:27.146767 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.128458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48064 (local address 127.0.98.61:43113)
0619 14:47:27.128557 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.128561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.128813 (+ 252us) server_negotiation.cc:407] Connection header received
0619 14:47:27.128849 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.128852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.128893 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.128957 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.129759 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.130468 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.131592 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.131928 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.132001 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:27.142361 (+ 10360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:27.142365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:27.142368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:27.142397 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:27.144266 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.144801 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.144805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.144807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.144891 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.145290 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.145294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.145295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.145658 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:27.145834 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:27.146478 (+ 644us) server_negotiation.cc:299] Negotiation successful
0619 14:47:27.146611 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:47:27.149995 23297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb6c7cd1cf840308e526575034f9993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } } peers { permanent_uuid: "368fb0a65263421dbec5c3498b152093" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } } peers { permanent_uuid: "7698476615364546a0c818170529903b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } }
I20260619 14:47:27.150276 23297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7698476615364546a0c818170529903b, State: Initialized, Role: FOLLOWER
I20260619 14:47:27.150571 23222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb6c7cd1cf840308e526575034f9993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } } peers { permanent_uuid: "368fb0a65263421dbec5c3498b152093" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } } peers { permanent_uuid: "7698476615364546a0c818170529903b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } }
I20260619 14:47:27.150648 23297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb6c7cd1cf840308e526575034f9993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } } peers { permanent_uuid: "368fb0a65263421dbec5c3498b152093" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } } peers { permanent_uuid: "7698476615364546a0c818170529903b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } }
I20260619 14:47:27.150769 23222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 368fb0a65263421dbec5c3498b152093, State: Initialized, Role: FOLLOWER
I20260619 14:47:27.151157 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:47:27.151077 23222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb6c7cd1cf840308e526575034f9993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } } peers { permanent_uuid: "368fb0a65263421dbec5c3498b152093" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } } peers { permanent_uuid: "7698476615364546a0c818170529903b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } }
I20260619 14:47:27.152107 23297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:27.152541 23121 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ebb6c7cd1cf840308e526575034f9993"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57050
I20260619 14:47:27.152804 23121 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 14:47:27.153316 23222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:27.153411 23121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:27.154935 23121 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb6c7cd1cf840308e526575034f9993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } } peers { permanent_uuid: "368fb0a65263421dbec5c3498b152093" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } } peers { permanent_uuid: "7698476615364546a0c818170529903b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } }
I20260619 14:47:27.156445 23196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ebb6c7cd1cf840308e526575034f9993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "368fb0a65263421dbec5c3498b152093"
I20260619 14:47:27.156682 23196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:27.158066 23196 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebb6c7cd1cf840308e526575034f9993 in term 1.
I20260619 14:47:27.158438 23095 leader_election.cc:302] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 368fb0a65263421dbec5c3498b152093, ebb6c7cd1cf840308e526575034f9993; no voters:
I20260619 14:47:27.158927 23121 leader_election.cc:288] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [CANDIDATE]: Term 1 election: Requested vote from peers 368fb0a65263421dbec5c3498b152093 (127.0.98.61:43113), 7698476615364546a0c818170529903b (127.0.98.60:40061)
I20260619 14:47:27.159345 23271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ebb6c7cd1cf840308e526575034f9993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7698476615364546a0c818170529903b"
W20260619 14:47:27.159807 23331 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 14:47:27.159807 23271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b [term 0 FOLLOWER]: Advancing to term 1
W20260619 14:47:27.159951 23331 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:47:27.161666 23271 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebb6c7cd1cf840308e526575034f9993 in term 1.
I20260619 14:47:27.162081 23154 master_runner.cc:186] Error getting master registration for 127.0.98.60:40061: OK, instance_id { permanent_uuid: "7698476615364546a0c818170529903b" instance_seqno: 1781880447085959 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:47:27.165354 23154 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:47:27.165472 23304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [term 1 FOLLOWER]: Leader election won for term 1
W20260619 14:47:27.166270 23347 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:47:27.166349 23347 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:47:27.168485 23228 master_runner.cc:263] No leader master found from master 7698476615364546a0c818170529903b
I20260619 14:47:27.168555 23228 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:47:27.168596 23304 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [term 1 LEADER]: Becoming Leader. State: Replica: ebb6c7cd1cf840308e526575034f9993, State: Running, Role: LEADER
I20260619 14:47:27.168959 23304 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb6c7cd1cf840308e526575034f9993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } } peers { permanent_uuid: "368fb0a65263421dbec5c3498b152093" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } } peers { permanent_uuid: "7698476615364546a0c818170529903b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } }
I20260619 14:47:27.170852 23304 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebb6c7cd1cf840308e526575034f9993. Latest consensus state: current_term: 1 leader_uuid: "ebb6c7cd1cf840308e526575034f9993" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb6c7cd1cf840308e526575034f9993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } } peers { permanent_uuid: "368fb0a65263421dbec5c3498b152093" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } } peers { permanent_uuid: "7698476615364546a0c818170529903b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } } }
I20260619 14:47:27.170972 23304 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:27.171757 23319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb6c7cd1cf840308e526575034f9993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } } peers { permanent_uuid: "368fb0a65263421dbec5c3498b152093" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } } peers { permanent_uuid: "7698476615364546a0c818170529903b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } } }
I20260619 14:47:27.171873 23319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:47:27.171643 23320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb6c7cd1cf840308e526575034f9993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } } peers { permanent_uuid: "368fb0a65263421dbec5c3498b152093" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } } peers { permanent_uuid: "7698476615364546a0c818170529903b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } } }
I20260619 14:47:27.174198 23320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:47:27.176362 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.162560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48074 (local address 127.0.98.61:43113)
0619 14:47:27.162673 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.162679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.165620 (+ 2941us) server_negotiation.cc:407] Connection header received
0619 14:47:27.165854 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.165859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.165915 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.165973 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.166980 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.167717 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.168912 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.169644 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.169721 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:27.171593 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:27.171597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:27.171600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:27.171627 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:27.173661 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.174951 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.174955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.174956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.175041 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.175347 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.175350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.175351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.175582 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:27.175687 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:27.176075 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 14:47:27.176200 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:47:27.177217 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.164245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.0.98.60:40061)
0619 14:47:27.164345 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.164349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.166707 (+ 2358us) server_negotiation.cc:407] Connection header received
0619 14:47:27.166982 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.166986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.167064 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.167119 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.169712 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.170404 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.171569 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.171895 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.171979 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:27.173595 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:27.173598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:27.173601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:27.173627 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:27.175302 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.175814 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.175817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.175819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.175919 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.176235 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.176240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.176242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.176454 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:27.176589 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:27.176938 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 14:47:27.177068 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:47:27.179761 23354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:27.181376 23354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:27.183702 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:27.162301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57054 (local address 127.0.98.62:44573)
0619 14:47:27.162495 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.162502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.165060 (+ 2558us) server_negotiation.cc:407] Connection header received
0619 14:47:27.165208 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.165212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.165265 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.165341 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.166352 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.167204 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.168346 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:27.168704 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:27.168779 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:27.179627 (+ 10848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:27.179631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:27.179635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:27.179667 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:27.181448 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.181997 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.182001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.182004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.182094 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:27.182395 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:27.182398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:27.182400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:27.182625 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:27.182748 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:27.183401 (+ 653us) server_negotiation.cc:299] Negotiation successful
0619 14:47:27.183551 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260619 14:47:27.192131 23271 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b [term 1 FOLLOWER]: Refusing update from remote peer ebb6c7cd1cf840308e526575034f9993: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:47:27.192543 23196 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093 [term 1 FOLLOWER]: Refusing update from remote peer ebb6c7cd1cf840308e526575034f9993: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:47:27.192941 23352 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7698476615364546a0c818170529903b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:47:27.193135 23304 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [LEADER]: Connected to new peer: Peer: permanent_uuid: "368fb0a65263421dbec5c3498b152093" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:47:27.199584 23320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebb6c7cd1cf840308e526575034f9993. Latest consensus state: current_term: 1 leader_uuid: "ebb6c7cd1cf840308e526575034f9993" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb6c7cd1cf840308e526575034f9993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } } peers { permanent_uuid: "368fb0a65263421dbec5c3498b152093" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } } peers { permanent_uuid: "7698476615364546a0c818170529903b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } } }
I20260619 14:47:27.199731 23320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:47:27.203280 23319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebb6c7cd1cf840308e526575034f9993. Latest consensus state: current_term: 1 leader_uuid: "ebb6c7cd1cf840308e526575034f9993" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb6c7cd1cf840308e526575034f9993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } } peers { permanent_uuid: "368fb0a65263421dbec5c3498b152093" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } } peers { permanent_uuid: "7698476615364546a0c818170529903b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } } }
I20260619 14:47:27.203423 23319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:47:27.207605 23352 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ebb6c7cd1cf840308e526575034f9993" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb6c7cd1cf840308e526575034f9993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } } peers { permanent_uuid: "368fb0a65263421dbec5c3498b152093" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } } peers { permanent_uuid: "7698476615364546a0c818170529903b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } } }
I20260619 14:47:27.207747 23352 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:27.207883 23355 mvcc.cc:205] Tried to move back new op lower bound from 7298582311691669504 to 7298582311607795712. Current Snapshot: MvccSnapshot[applied={T|T < 7298582311691669504}]
I20260619 14:47:27.210481 23319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ebb6c7cd1cf840308e526575034f9993" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb6c7cd1cf840308e526575034f9993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } } peers { permanent_uuid: "368fb0a65263421dbec5c3498b152093" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } } peers { permanent_uuid: "7698476615364546a0c818170529903b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } } }
I20260619 14:47:27.210613 23319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:47:27.211825 23358 mvcc.cc:205] Tried to move back new op lower bound from 7298582311691669504 to 7298582311607795712. Current Snapshot: MvccSnapshot[applied={T|T < 7298582311691669504}]
I20260619 14:47:27.213863 23304 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ebb6c7cd1cf840308e526575034f9993" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb6c7cd1cf840308e526575034f9993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } } peers { permanent_uuid: "368fb0a65263421dbec5c3498b152093" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } } peers { permanent_uuid: "7698476615364546a0c818170529903b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } } }
I20260619 14:47:27.214001 23304 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:27.214944 23320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ebb6c7cd1cf840308e526575034f9993" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb6c7cd1cf840308e526575034f9993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44573 } } peers { permanent_uuid: "368fb0a65263421dbec5c3498b152093" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43113 } } peers { permanent_uuid: "7698476615364546a0c818170529903b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40061 } } }
I20260619 14:47:27.215075 23320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:47:27.219571 23356 mvcc.cc:205] Tried to move back new op lower bound from 7298582311691669504 to 7298582311607795712. Current Snapshot: MvccSnapshot[applied={T|T < 7298582311691669504}]
I20260619 14:47:27.220146 23354 catalog_manager.cc:1347] Generated new cluster ID: a500ef8977584ea99d5c78d746490187
I20260619 14:47:27.220222 23354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:27.247923 23354 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:27.249459 23354 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:27.281239 23354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebb6c7cd1cf840308e526575034f9993: Generated new TSK 0
I20260619 14:47:27.282230 23354 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:47:27.321328 23150 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:27.318255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57066 (local address 127.0.98.62:44573)
0619 14:47:27.318385 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.318390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.320240 (+ 1850us) server_negotiation.cc:407] Connection header received
0619 14:47:27.320433 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.320436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.320489 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.320551 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.321184 (+ 633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57066: BlockingRecv error: recv got EOF from 127.0.0.1:57066 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 14:47:27.322649 23226 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:27.318990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.0.98.61:43113)
0619 14:47:27.319105 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.319109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.321813 (+ 2704us) server_negotiation.cc:407] Connection header received
0619 14:47:27.321984 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.321987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.322035 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.322084 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.322531 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48086: BlockingRecv error: recv got EOF from 127.0.0.1:48086 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 14:47:27.324297 23300 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:27.319644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58600 (local address 127.0.98.60:40061)
0619 14:47:27.319787 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.319792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.323149 (+ 3357us) server_negotiation.cc:407] Connection header received
0619 14:47:27.323542 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.323546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.323601 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.323657 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.324189 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58600: BlockingRecv error: recv got EOF from 127.0.0.1:58600 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 14:47:27.328962 23226 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:27.327338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48088 (local address 127.0.98.61:43113)
0619 14:47:27.327515 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.327520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.327943 (+ 423us) server_negotiation.cc:407] Connection header received
0619 14:47:27.328178 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.328182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.328259 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.328314 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.328852 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48088: BlockingRecv error: recv got EOF from 127.0.0.1:48088 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 14:47:27.334285 23226 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:27.332664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48098 (local address 127.0.98.61:43113)
0619 14:47:27.332766 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.332770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.333292 (+ 522us) server_negotiation.cc:407] Connection header received
0619 14:47:27.333480 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.333484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.333538 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.333593 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.334183 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48098: BlockingRecv error: recv got EOF from 127.0.0.1:48098 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 14:47:27.340481 23226 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:27.338818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48104 (local address 127.0.98.61:43113)
0619 14:47:27.338934 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.338939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.339498 (+ 559us) server_negotiation.cc:407] Connection header received
0619 14:47:27.339715 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.339719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.339775 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.339827 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.340371 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48104: BlockingRecv error: recv got EOF from 127.0.0.1:48104 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 14:47:27.352829 23226 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:27.347915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48114 (local address 127.0.98.61:43113)
0619 14:47:27.348156 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.348163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.349235 (+ 1072us) server_negotiation.cc:407] Connection header received
0619 14:47:27.349497 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.349501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.349585 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.349647 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.352678 (+ 3031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48114: BlockingRecv error: recv got EOF from 127.0.0.1:48114 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260619 14:47:27.366556 23226 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:27.364320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48128 (local address 127.0.98.61:43113)
0619 14:47:27.364468 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.364474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.365243 (+ 769us) server_negotiation.cc:407] Connection header received
0619 14:47:27.365493 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.365497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.365565 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.365651 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.366425 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48128: BlockingRecv error: recv got EOF from 127.0.0.1:48128 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 14:47:27.387097 23226 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:27.385774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48132 (local address 127.0.98.61:43113)
0619 14:47:27.385915 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.385921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.386132 (+ 211us) server_negotiation.cc:407] Connection header received
0619 14:47:27.386270 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.386274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.386336 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.386444 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.386962 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48132: BlockingRecv error: recv got EOF from 127.0.0.1:48132 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 14:47:27.422955 23226 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:27.421503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48142 (local address 127.0.98.61:43113)
0619 14:47:27.421685 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.421690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.421853 (+ 163us) server_negotiation.cc:407] Connection header received
0619 14:47:27.422048 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.422053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.422172 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.422279 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.422791 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48142: BlockingRecv error: recv got EOF from 127.0.0.1:48142 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 14:47:27.491185 23226 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:27.489603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48158 (local address 127.0.98.61:43113)
0619 14:47:27.489838 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.489875 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.490134 (+ 259us) server_negotiation.cc:407] Connection header received
0619 14:47:27.490342 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.490345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.490397 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.490501 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.491029 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48158: BlockingRecv error: recv got EOF from 127.0.0.1:48158 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260619 14:47:27.623333 23226 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:27.621822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48162 (local address 127.0.98.61:43113)
0619 14:47:27.622036 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.622040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.622065 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:47:27.622289 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.622292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.622348 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.622450 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.623077 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48162: BlockingRecv error: recv got EOF from 127.0.0.1:48162 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 14:47:27.883201 23226 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:27.881859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48164 (local address 127.0.98.61:43113)
0619 14:47:27.882282 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.882286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.882300 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:47:27.882351 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.882353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.882406 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.882482 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.883064 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48164: BlockingRecv error: recv got EOF from 127.0.0.1:48164 (error 108)
Metrics: {"server-negotiator.queue_time_us":256}
I20260619 14:47:28.161518 23331 catalog_manager.cc:1259] Loaded cluster ID: a500ef8977584ea99d5c78d746490187
I20260619 14:47:28.161605 23331 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093: loading cluster ID for follower catalog manager: success
I20260619 14:47:28.163306 23331 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093: acquiring CA information for follower catalog manager: success
I20260619 14:47:28.164610 23331 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 368fb0a65263421dbec5c3498b152093: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:47:28.167428 23347 catalog_manager.cc:1259] Loaded cluster ID: a500ef8977584ea99d5c78d746490187
I20260619 14:47:28.167582 23347 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b: loading cluster ID for follower catalog manager: success
I20260619 14:47:28.169246 23347 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b: acquiring CA information for follower catalog manager: success
I20260619 14:47:28.170403 23347 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7698476615364546a0c818170529903b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:47:28.402117 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:28.397665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48170 (local address 127.0.98.61:43113)
0619 14:47:28.397941 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:28.397948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:28.397968 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:47:28.398226 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:28.398230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:28.398372 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:28.398509 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:47:28.398802 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:28.399390 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:28.400455 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:28.400718 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:28.400842 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:28.401199 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:47:28.401673 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:47:28.401822 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:28.401867 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 14:47:28.401934 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":70,"threads_started":1}
I20260619 14:47:28.409685 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:28.405632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58604 (local address 127.0.98.60:40061)
0619 14:47:28.405880 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:28.405887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:28.406036 (+ 149us) server_negotiation.cc:407] Connection header received
0619 14:47:28.406260 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:28.406270 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:28.406390 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:28.406556 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:47:28.406788 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:28.407305 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:28.408182 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:28.408412 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:28.408529 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:28.408828 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:47:28.409286 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:47:28.409422 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:28.409468 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 14:47:28.409523 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
I20260619 14:47:28.417778 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:28.413142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48180 (local address 127.0.98.61:43113)
0619 14:47:28.413319 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:28.413323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:28.413882 (+ 559us) server_negotiation.cc:407] Connection header received
0619 14:47:28.413941 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:28.413945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:28.414032 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:28.414120 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:47:28.414503 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:28.415320 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:28.416447 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:28.416829 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:28.416952 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:28.417225 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:47:28.417313 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:47:28.417387 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:28.417564 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 14:47:28.417634 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 14:47:28.417798 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:28.413819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58618 (local address 127.0.98.60:40061)
0619 14:47:28.414090 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:28.414095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:28.414110 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:47:28.414155 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:28.414158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:28.414204 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:28.414290 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:47:28.414614 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:28.415358 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:28.416520 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:28.416804 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:28.416877 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:28.417299 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:47:28.417385 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:47:28.417459 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:28.417632 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 14:47:28.417685 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260619 14:47:28.417801 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:28.413085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57078 (local address 127.0.98.62:44573)
0619 14:47:28.413372 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:28.413376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:28.413393 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:47:28.413491 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:28.413494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:28.413669 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:28.413811 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:47:28.414065 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:28.414827 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:28.415997 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:28.416349 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:28.416508 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:28.416820 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:47:28.417252 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:47:28.417333 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:28.417526 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 14:47:28.417591 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
I20260619 14:47:28.421826 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 23079
I20260619 14:47:28.428306 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 23154
I20260619 14:47:28.434432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ9bYsM/build/debug/bin/kudu with pid 23228
2026-06-19T14:47:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3186 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6440 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (293167 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST